Update
It looks like several things changed with some latest changes (October 7th) in the develop branch. It now looks like it's possible to support a module living in the vendor directory.
I created two examples of installing modules. One that copies files to app/code and another that registers the module where it resides in the vendor directory.
Copy Strategy: https://github.com/mttjohnson/magento2-sample-module-minimal-copy Registration Strategy: https://github.com/mttjohnson/magento2-sample-module-minimal-register
For development purposes I think utilizing the registration strategy is more useful because the files that are being run by Magento can be directly modified, tested and then committed back to the module repo.
Original Answer
If you have a composer.json file for your module composer package that contains a type of magento2-module then by default files will get copied over to a specified mapped directory in the app/code directory.
{
"name": "vendorname/module-name",
"type": "magento2-module",
"require": {
"magento/magento-composer-installer": "*"
},
"extra": {
"map": [
[
"module",
"VendorName/ModuleName"
]
]
}
}
In this example composer.json the composer package name vendorname/module-name will result in the files for the composer package getting placed in vendor/vendorname/module-name.
The special type of magento2-module is implemented as a composer-plugin in the magento/magento-composer-installer package. That is why I have listed it in the require section. It is this composer-plugin that does the copying of files over into the appropriate magento app/code directory.
The extra: {map: [["composerDir","MagentoDir"]]} section is referenced by the composer-plugin installer to know what part of your composer package to map to what part of your magento directory structure. In the example provided this would take vendor/vendorname/module-name/module and copy files from there to app/code/VendorName/ModuleName.