The Metasploit Framework has grown in leaps and bounds: what used to be a small team of free-time developers is now an actual product team working for a real company. The community that contributes to the open source framework has continued to expand; instead of a few of active contributors, we now have over a dozen, not counting all of the drive-by-coders that submit patches and modules through the Redmine tracking system. As the code base grows, so does our user base, and quality has become the most important feature of the product.
The huge number of contributions on one hand, and the quality demands on the other, have lead to a situation where our existing development process was starting to feel the strain. To keep up with the load, we have made major investments in our automated quality testing. Continous integration tools, automated lab management APIs, and other changes to the code base have helped us keep up with the increasing size of the project. The development tools, however, were still significantly lacking.
I'll admit it. I used CVS and Bugzilla until 2005. I have been using Subversion and Trac/Redmine ever since. The latest and greatest in source code management tools has always seemed less interesting to me than the actual code I was working on - if the solution worked, there wasn't any strong reason to move away from it. We have finally reached the point where a centralized development model we have today was making it hard for folks outside the project to easily contribute. Even folks who had direct commit access could use a better branching model and testing process before pushing those changes back into the development trunk.
All development on the Metasploit Framework (along with warvox, msfrpc-client, and nexpose-client) is now managed though the Git SCM and the GitHub.com development portal. This is is a major change for a few reasons:
Anyone can fork the official repository, modify the code, and send a pull request. All existing developers will use this exact model. This makes contributing a module and getting peer review nearly painless.
Anyone can fork the offiical repository, add their own code to this fork, and make this available to other Metasploit Framework users. GitHub provides both Git and Subversion access to all hosted respositories, including forks like this. Switching to an experimental branch takes three commands:
# cd /opt/metasploit-4.1.4
# mv msf3 msf3.official
# svn co https://svn.github.com/
All merges to the official repository are signed off by a Rapid7 employee. This may seem restrictive, but the goal is to increase accountability and prevent dumb bugs from making their way into the code base. This code review happens today, but bugs can still slip past when a developer merges an external patch without adequate review. Git's branching model and pull request system will make this process painless and transparent. This overhead is balanced by the fact that any developer can publish their personal fork while their changes go through code review.
This changes the development process, but keeps the existing Subversion update mechanism, along with msfupdate and other svn-based tools working as-is. This was accomplished through a pile of custom code currently named "Charon"; merges to the master branch in GitHub are replicated to our existing Subversion repository, which is mirrored through the standard https://metasploit.com/svn/framework3/trunk/ URL. If you use Subversion today to keep updated, keep using it. This architecture lets us introduce additional quality checks between merges to Git and their replication to the Subversion trunk and will not be going away anytime soon.
A huge thanks to Matt Buck and Trevor Rosen who gave up any pretense of a work-life balance for the last week to make this happen. This is one of many changes coming to the project to expand opportunities for collaboration while continuing to raise the quality bar.