If you look at attackers as faceless, sophisticated digital ninjas, it instills fear, but doesn't really help to stop them. While there are many motivations for attacking an organization and stealing its data, the most frequent are based on money. This is why it sometimes helps to view them as you would any other business: as having costs and needing to generate revenue to survive.
Attacker groups are similar to high-tech startups
There is a thriving economy full of people who breach organizations, steal the information contained on their systems, and sell it. There are teams that cover the breadth of skills necessary and those who specialize on social engineering, exploit development, or drop servers, but the consistency across them all is that they will cease to exist if they cannot fund their efforts. The most threats to the vast majority of organizations today is not going to be funded by a foreign government; they need to successfully sell information to continue paying social engineers or buying credentials, servers, exploits, malware, and any other tools they need to steal from their next target.
The most successful groups are continually building new tools, testing their effectiveness and iterating on their process according to the results. They learn from any failed attempts to gain access to a network, move through the organization, and exfiltrate data they can sell. This process of trying, learning from the results, expanding what works and throwing away what doesn't is very similar to the Lean Startup methodology. A lot of businesses have developed their "secret sauce" through this process and criminal groups are similarly finding their "secret sauce" of tools they use to steal and sell information without going to prison.
Financially motivated attackers increase their revenue through continued access
There is one area in which these groups have not been forced to iterate a great deal. In the last few years, the breaches yielding the most sales for their crew of attackers used a lot of very similar tools to, once inside, steadily explore the network undetected. "Smash and grab" attacks still happen, but to really maximize their return, attackers need to gain access to the network and progressively access more systems to find the valuable data they can sell to others, whether it is in the form of credit card data, personal information, health records, or intellectual property. They make no money from the initial compromise, so they are incented to stay inside undetected for as long as possible and the results are on their side. Though it does take them time to make money from a breach, it takes an order of magnitude more time for most organizations to detect them.
If you have pets, as I do, this description might conjure images of a tick. I am not going to include a real image of these disgusting parasites because I want someone to read this post, but I see a great deal of similarities and we can learn a little from our experience dealing with them. We cannot completely eliminate ticks, but they also don't really benefit, or cause significant harm, unless they remain on you or your animal undetected for a long period of time. For this reason, as ticks with a resistance to prevention-only tick treatments survived and multiplied [yay, natural selection!], a new chemical was added to slowly kill them from the bite because it was no longer assumed you could just stop them all from latching. Even with these treatments, we still need to periodically check for ticks because resistant ones emerge and finding them before they do damage is so important.
We need to significantly shorten the time to contain to cut off their revenue
Just as letting a tick live on your household pet for more than a day can lead to both a satisfied parasite and serious disease, letting intruders remain on your network for more than a few hours means money for the attacker and damage to your organization. Unfortunately, there is no "treatment" we can give our internal systems to kill attacker hard drives, but when we accept that some attackers will get onto our network eventually, we can focus on detecting them once inside and stopping them from stealing any valuable data. Finding and eliminating the parasitic intruders soon after the initial compromise has the same impact on their ability to survive as a business that effective prevention does. While there are many reasons startups fail, the consistency across the failures is the inability to function without revenue.
To learn more about Rapid7's Incident Detection and Response solutions, check out our new solutions page which includes our Incident Response Services.