If You Work In Operations, Your Security Team Needs The Logs, Too

Last updated at Mon, 28 Oct 2019 17:17:00 GMT

This post is the final in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the previous six, click one, two, three, four, five,or six.

As a final discussion topic around search and analytics, it is really important to talk about the different teams involved, who all have different questions to ask of the same overall data. Roles and titles can vary greatly, but for simplicity, I'll refer to the three primary groups as ‘IT Ops' [for anyone looking to maintain servers and software required for the employees and partners of the organization], ‘DevOps' [for anyone focused on the availability and quality of servers and software for external customers], and ‘security' [for the individuals looking to prevent and detect malicious activities]. Let's talk about the order in which these teams generally develop in an organization and how that leads to politics between teams and the duplication of efforts, using the now-classic Lethal Weapon series.

IT operations monitored logs first, but not last

For our purposes here, I'm going to consider the IT Ops team the Roger Murtaugh of the company because they were likely there first, so they were around to acquire the necessary tools and lay out “the way things are done”. When the first email servers were stood up and various infrastructure deployed, IT Ops were manually checking the health of each system and receiving floods of automated emails with complex error codes when the slightest problem inevitably occurred. This was never an easy job, but the gradual addition of ticketing systems, helplines, email distribution lists, and pagers to their processes made it possible to handle the day-to-day needs and off-hours emergencies. Centralized log management solutions were then brought in to simplify the entire monitoring and troubleshooting process, and the first automated trend charts around CPU usage, errors, and bandwidth made drastic improvements on initial notifications. Troubleshooting could finally be done from a single place, so IT Ops was happy with the way things were working.

If your organization has custom-developed software, as many do today, at least one DevOps team has gotten sick of similarly SSHing into production systems to verify a new release is functional or track down a bug. For the IT Ops team, realizing this new team is looking to implement a log management solution was probably a similar shock to when Murtaugh first learned he had to work with that loose cannon, Riggs, who did things his own way. Given that change can be scary, providing high-level permissions to access your prized tools and letting another team do things “their way” both feel inherently risky. Even once these teams come to an agreement to either share a solution or work independently, they are typically minimizing very specific risks to the business with little thought to the risks with which the security team must obsess.

You're mistaken if you've assumed no one else needs your logs

One of my favorite cognitive biases to consider, because we can all rarely avoid it, is the "focusing illusion" best summarized as "nothing in life is as important as you think it is while you are thinking about it". This is relevant here because when you are paid, trained, and thanked for using the tools and data at your disposal for very clear purposes, you are unlikely to realize just how much value is in there for others with very different questions or that sometimes those other questions could be more important to the company than your own. Whether or not Murtaugh and Riggs in your organization have become partners, they most likely see security as a pest [like Leo Getz] or like Internal Affairs [Riggs's future wife, Lorna Cole] because the effort to reduce the risk of attacks and data leaks often requires very different efforts than keeping applications functional and available for customers, be they internal or external. Meanwhile, in reality, just as all of these characters had the same goal [of stopping serious criminals], the security team has the same overarching goal as IT Ops and DevOps: eliminating reasons the rest of the company would be unable to accomplish its goals [and yes, I remember that Leo Getz was originally a money-launderer]. If you're in IT Ops or DevOps, your security team wants you to know that they would really like access to your logs. It would help them reduce the risks they worry about, even if you don't completely understand why.

It's healthier not to differentiate between “logs” and “security logs”

Microsoft may have biased us all by labeling some logs as "security logs" and others for different purposes, but the easiest example of non-security logs with value to security are DHCP logs; if your security team doesn't have access to DHCP lease grants and releases, it can be very difficult to find out which machine is operating at any given IP address in an incident investigation. The same goes for machine data the DevOps team considers only relevant to them - if you can see who made the last change to the production servers before they went down, you can also see whose account might have been stolen and used to change settings preventing outside access to production systems. If you never ask the security team if your logs would be useful, you are likely to make the wrong assumptions. Besides, they should absolutely have the security tools capable of ignoring anything they don't need.

Getting a single solution for all parties to use can save you a lot

Though the security team should have the tools that help them find what they care about in your logs, far too many organizations have three different tools for three different teams when there could be a single one. You don't need pivot tables or fancy finance visualizations to realize that buying a log management solution for IT Ops, a separate search solution for DevOps, and also a SIEM for the security team is a very costly way to solve different problems with the same data. Even if you forget about the three different purchases, failing to take advantage of discounted tiers of pricing, you can see that three different groups have to manage three different solutions focused on search and analytics which only differ by the questions they ask and use cases to which they apply. There may have been technology challenges preventing this before, but you should demand better now. Ask for real-time search and analytics flexible enough to satisfy Murtaugh's, Riggs's, Getz's, and Cole's problems. You all want to analyze your data to remove anything which could possibly hinder your organization's progress.

If you are a part of the IT Ops or DevOps teams, you can start a free trial of Logentries here and search your data within seconds.If you are on the security side of things, check out an InsightIDR demo.