Search Will Always Be A Part of Incident Investigations

Last updated at Mon, 28 Oct 2019 17:09:20 GMT

This post is the first in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle.

Strong data analytics have recently enabled security teams to simplify and speed incident detection and investigation, but at some point of every incident investigation, a search through machine data is nearly always necessary to answer a one-time question before the investigation can be closed. Whether your incident response team is just trying to combat the flood of alerts coming from a series of noisy detection appliances or seasoned specialists who have automated a sizable amount of data enrichment to focus on manually hunting the sophisticated, there are many reasons you need the ability to search through disparately sourced machine data for answers.

Attackers try for easy, then adapt and iterate

For those who mostly hear about massive data breaches via the major media outlets prior to the substantive details being publicized, hindsight bias brings a great deal of thoughts like "they should have known to watch their medical record database more closely" and "how could they have not watched the applications on POS devices better?", but this is drastically over-simplifying the attacks, themselves. Every attack plays out as the intruders adjust to the preventive controls in place. Stealing partner credentials was likely not the first technique attempted for initially gaining access to the Target network, but it was the first to work. Though the initial compromise typically occurs via either stolen credentials or web application exploit, the next step taken will be the same isolated trial-and-error process of trying some basics like harvesting more credentials and scanning for potential new hosts or a five year-old remote exploit, yet it is guided by what works in that one moment in time.

If you've watched Braveheart as many times as I have [and you probably haven't], you may see the similarities between its [historically inaccurate] battle tactics and a cyber-attack. Whether it was luring a large number of soldiers into a gully before revealing your army on the cliff above, pretending to have no plan for the English cavalry, or a gentlemen's agreement with the onrushing Irish, every skirmish was started and progressed in a different way. Being crafty doesn't always lead to success, but it leads to a lot more success than quitting after trying the easiest tactic and failing.

This unpredictable flow of every attempted compromise is why organizations so often don't know what data they should be collecting for investigative reasons. There will always be high value data sources in organizations which provide necessary context for detection and investigation, but they will have limitations when attackers adapt to what they successfully compromise. Analytics just won't help explore the data which may only be relevant to an incident investigation one time.

Every organization and corresponding environment is unique

It is frequently underestimated just how organically an every network grows, but as evidenced by the recent report on the IRS feeling lost to upgrade its Windows systems, it is extremely challenging just to identify every endpoint and server with access to your network. Add to that just how easy it is for software developers to spin up a small cloud environment or a financial analyst to take work home on an iPad, and you start to see the attack surface expand rapidly.

If you are limiting the data you collect for monitoring and investigation to what is easily centralized or on the managed perimeter, your incident response team is forced to close a great deal of incoming alerts and follow-up investigations without the high level of confidence they would prefer. They simply don't have enough of the data they would need to know what happened.

Custom application logs and unstructured evidentiary data are needed for confident incident analysis

Of even greater concern than the large blind spots in the structured data is the unstructured data often unavailable to security teams. If your organization has a great deal of internally-developed applications, as most modern businesses do, and you're only monitoring the logs from your firewall, domain controller, and web proxy, you'll be blind to an attacker moving from server to server in your production environment, looking for some data to monetize. For those who think this is rare, our Logentries team estimates that thirty percent of the logs we process for our customers are completely custom. In these scenarios in which the relevant data to completing incident analysis could be unstructured and unique to your company, your security team needs the ability to search it quickly for data points which tie it to the broader investigation. Inflexible log management solutions are of no help here.

Understanding the root cause can take many forms

What incident response teams rarely say, but typically know, is that the major reason incident management is so difficult is that you're using machine data generated as a result of some actions and working to determine exactly what an actor, be it human or software, did to cause this resulting data generation. Then, you not only have to understand the behavior, but also have a high enough level of confidence to explain the actor's intentions to either take action (when malicious) or close the investigation (when benign). Every person and team has its own comfort level, but the process to reach it is different for every type of initial behavior and too often isn't reached because of insufficient data or the inability to properly explore it in a timely fashion.

If you really want to understand the root cause and close your investigations at a pace to avoid alert fatigue, you need the context around user and endpoint behavior you can only obtain from an analytics solution plus the flexibility and ease of machine data search solutions. You cannot limit yourself to one or the other and maintain that high level of confidence. If this sounds interesting to you, we do have a number of Incident Detection & Response solutions that you can read about here.