Last updated at Thu, 26 Oct 2017 19:11:30 GMT
At Logentries we process over 10 billion log events every day. That’s quite a lot of data from quite a lot of systems…all being processed and analyzed by our cloud service. We realized that this puts us in a unique position to look at a huge amount of data from a macro level and provide our community with insights as to what is happening across different platforms and software components (of course, all anonymized with privacy protected).
So our Logentries research team decided to take advantage of this unique position and set out to examine a sample of our overall user base to provide some initial insights. More precisely, we analysed approximately 22 billion log events from more than 6,000 Heroku applications from the perspective of someone in a DevOps role.
Many of our Heroku end users have a DevOps role within their organizations, where they are the developers of the system and they’re also responsible for operations when the system goes live – very often the main reasons they choose Heroku & Logentries in the first place was to ease their operational responsibilities. Someone building and managing a Heroku application is usually pretty interested in the performance and reliability of their service. And they will often spend time digging for errors or application exceptions in their logs during development debugging or when there is a production issue that needs to be resolved.
Thus we decided that ‘relevant events’ for someone in this DevOps role could be defined as all Heroku error codes and application exceptions. Taking this definition of ‘relevant events’ we analyzed a sample of 22 billion log events from a section of our Heroku user base and found that a very small % of the overall log data was interesting for this particular use case. In fact, from our research, we found that less than 0.18% of the 22 billion log events were errors or application exceptions – so 99.82% was signal noise crowding out the important information if you are debugging or troubleshooting.
That’s not to say that that 99.82% of log data is useless – on the contrary, in fact, it is a constant stream of data flowing from your systems with a lot of valuable info – and it usually contains valuable data for security, compliance and business use cases. For example, how many people visited your site yesterday, how many signed up, how many converted? However, our hypothesis is, for any given use case there’s generally a lot of signal noise. In other words, the other useful data gets in the way sometimes and you need to filter this out for a particular task. This is exactly what we found for the DevOps use case and argues the case for the need to effectively manage your log data so you can separate the little data that matters for you right now, from the large body of other information contained in your logs.
You can read more on what we found from analyzing this data in our full research report: “Big insights from Little Data: A Spotlight on Unlocking Insights from the Log Data That Matters.”
To read the rest of the research done by the Logentries research team, check out the research section of Logentries Insights.
