Last updated at Tue, 12 Dec 2023 18:17:59 GMT
This is a guest post from a long-time Metasploit contributor and community member. Over the next few months, Rapid7 will be publishing a series of guest posts featuring unique perspectives on Metasploit Framework and highlighting some of our community’s favorite functionality, hidden gems, and backstories.
Want to contribute an idea or a post? Reach out to community[at]rapid7.com.
Back in my day, you could get dinner, dessert, and ride the trolley home all for a nickel. Oh, and we used SVN for source control and Redmine for issue tracking, our shells were clear-text, and shikata ga nai really "couldn't be helped" by inspection engines faced with trying to decode it in flight…
Since I got involved with the Metasploit Project a decade ago, it has been adopted by a commercial steward; bloomed with modules, documentation, and internal capability; and spilled gallons of precious developer brain matter enforcing convention and standards to ensure the project's ability to scale and function as the underpinning of a product with all the wonders of support contracts attached to it. We've seen great contributors come and go, leaving behind them innovative code, modules, and core functionality for the rest of us to use and build upon—a base that lets us do everything from actively interrogating service endpoints and passively listening for valuable data to giving us shells in fun new ways. We've had CVEs written on us—a reminder to all that what we do does not make us immune in the least from being the vector (if anything we're sweeter targets than any sysadmin)—and we've had heartache, regressions, and facepalms so loud we've woken our spouses, kids, and dogs more than once. All of this has been happening at the far corners of the globe and in your own backyard among hundreds of contributors, staff, and community members working together to design and build the most comprehensive open source penetration testing framework on Earth, full stop.
To understand what that actually means, we have to be able to answer a fundamental question: What is a penetration testing framework? Is it a sexy text console with archaic commands that gives us root shells, like we see in the movies? Is it a piece of software that somehow defies the rules and lets us pull unicorns out of thin air to autopwn the Gibson (credit to mubix for showing me the movie)? Is it a collection of scripts you can run to get information about something, or possibly to exploit a well-known problem?
A penetration testing framework can be used for all of that, but what defines a penetration testing framework is the capacity to provide malleable attacker controlled infrastructure as an entry point to the engagement. To the degree that it is possible, a penetration testing framework provides granular controls for context, data operations, communication, and execution to the attacker such as to give them the control and agility required to effect their intended action and rapidly achieve the objective, without being caught. It is the digital skin we use to touch the world we work in without leaving fingerprints behind or casting a shadow below, if we know what we're doing.
On a basic level, offensive operations require the ability to perform reconnaissance and effect change in a subject domain in order to achieve an objective. Since target objectives often reside on remote systems, reaching the subject domain requires the capacity to interact via networks with services using standards which govern the communication and interface aspects of the interaction. "Normal" software components (hopefully) work within the relevant set of operational constraints to perform the required communication and interface operations in order to execute a well defined set of application functions. Attackers looking to gain leverage over the involved components need much more flexibility than the standard client would provide so as to be able to alter the sequence of operations, data elements in the transaction, or aspects of the network communication itself (slow loris attacks are fun, right?). Attackers who don't want to get caught also want to ensure that they don't leak any attributable information, and that they maintain wire-level entropy levels sufficient to avoid flagging the client's NIDS. Practically speaking, the functional domain encapsulating the process of building a "normal" layer 7 interface is handled by both the OS and client software, meaning that granular control over the entire stack requires having to implement kernel and application functions from scratch in the framework...
The framework is, among other things, an attacker-controlled OS abstraction providing primitives for execution, protocol, socket, and frame operations that give the attacker unprecedented control over inputs and outputs—as well as being a one-size-fits-all application/protocol client and server.
At the heart of Metasploit is Rex: the Remote Exploitation Library. Rex is effectively the kernel of the framework, which provides the implementations and abstractions of these complicated functions to the rest of Framework. Over the years, the library has grown so much we've had to start extracting it to separate gems just to preserve our sanity. Using versioning in releases keeps breaking work-in-progress changes from causing massive havoc (I've broken the master branch more times than I can count offhand).
Rex deals with everything from Event and Socket primitives to parsing and encoding in other high-level languages (like PowerShell), all encapsulated in neat Ruby objects and modules. The primitives defined in Rex are used as building blocks in the Msf:: namespace mixins to provide the functions they encapsulate in a flexible organization structure that creates the user-facing Metasploit modules (who elected to call them modules?). Because the contributor community’s efforts toward full code discipline are still ongoing, these guidelines are not fully implemented rules: sometimes code that should be written in an encapsulated and abstracted manner down in Rex is written in the Msf:: namespace and needs to be moved down, or vice versa. Most of my work and knowledge of the Framework and its siblings is here, in /lib, where anything is possible and we are never even close to being “done.”
You may know me as RageLtMan, or Semper Victus if you watch the issue tracker/PR queue. I have a "bit" of opinionated effort invested in the project. Significant segments of those efforts are accepted upstream, often in frequently-executed junctures of the code base. The Rapid7 team has graciously extended an offer to let me run my mouth once in a while about the murky depths of things you won’t find in the quickie how-to-use-it guides, or unfortunately sometimes even in the documentation. Over the coming months I plan to do my best to explain how some of the inner magic such as socket pivoting and transport encapsulations work, how to write "friendly framework" code in /lib which gets iterated on ad nauseum, and how to actually put some of the tooling submerged here to practical use on engagement.
With everything the community has taught me over the years, I hope to use this opportunity to give back in a new and meaningful way. That said, we want to hear from you, the reader, the user, the contributor: what parts of this giant clockwork orange do you want to read about, to better understand, or to improve? The more we all know about the core and its capabilities, the more we can do on engagement and in our research. So please comment or write in with questions and suggestions, find us on IRC [ed: or Slack!], or toss off an email with your thoughts.
This concludes my hello world rant. Please tune in next time for "The taste of our dogfood," or, more aptly, "maintaining security posture in our work as attackers."
-RageLtMan
