Last updated at Mon, 06 Nov 2017 18:56:47 GMT
In our January webinar, broadcasted & recorded on January 28th 2016, we shared 5 best practices for log management & analytics in 2016 that were inspired by the most common questions we received in 2015.
Below is a brief summary of those 5 best practices (plus one bonus practice). To watch a recording of the full webinar on-demand, click here. Also, be sure to register for February’s webinar on Monitoring Business-Critical Events, co-hosted with VictorOps.
Best Practices for 2016
1. Start with a strategy for where you’ll log from
It can be tempting to start logging from every asset available. Doing so comes with certain advantages, like being able to refer back to information you never expected to need in the future. Logging everything can be understandably overwhelming though and like most things, it’s important to have a strategy.
To help provide some direction for where to log from, we’ve put together a simple list for where to start. Details surrounding each section can be found in the full version of the webinar.
2. Log with unique identifiers
Logging with unique identifiers is the key to investigation & correlation.
Consider the example on the right – someone investigating a possible attack can identify a single suspicious action taken by a user and correlate that user ID with other actions related to updated privileges to identify other possibly impacted assets. None of this would be possible if unique identifiers weren’t included in the log event.
3. Log in structured or semi-structured format
A common best practice when logging is to log in “structured” format whenever possible. Examples of structured formats include key-value pairs and JSON. Two big advantages to logging in structured formats include:
- The data is easily human-readable by anyone looking at it
- The data can be aggregated by log management tools and thus used to calculate values like SUM, COUNT, AVERAGE, etc.
Not all services will log in structured formats though, so Logentries recently released support for automatically parsing semi-structured data. One of the most popular log formats used today by services like Apache web servers is Common Log Format. While Common Log Format doesn’t include key-value pairs as traditional structured data would, the order of details in a Common Log Formatted log is always the same. Logentries now automatically applies keys to each bit of information in a Common Log Formatted log, making it easier for a wider audience to analyze their log data.
4. Eliminate noise using percentiles
We found that in 2015, calculating the average of a dataset was one of the most popular queries executed in Logentries. Despite the popularity of AVERAGE, outliers in a dataset can skew an average, providing misleading results. To address this issue, Logentries recently introduced percentiles to help users eliminate misleading outliers and focus on more normalized data. We produced a helpful video explaining outliers here, or you can watch the full webinar for a recap.
5. Make alerts actually useful with thresholds, negative matching & more mediums
Too often, alerts can become mindless noise that goes ignored, making it unlikely for teams to appropriately react when important alerts do occur. In an effort to combat this issue, you should always ask the following questions when creating an alert:
- Do I really want to be alerted every time this event occurs? Would I only care if this event were to occur at a certain frequency within a particular timeframe? (If so, consider setting alerts with thresholds).
- If I expect an event to occur at a particular interval (like a cron job), do I really need to be alerted every time it occurs? Would it be more helpful to receive alerts only if this event does NOT occur as expected? (If so, consider negative-matching alerts, or “Inactivity Alerts”).
- Is email really the best tool to deliver alert notifications to my team? If my team spends most of their day in a tool like Slack rather than email, shouldn’t I send my alerts there? (if so, consider a tool like Logentries that offers direct integrations with a variety of different popular services.)
Bonus: Collaborate & plan, early & often
In an effort to practice good DevOps, your logging practices should be communicated and debated amongst all key stakeholders, including engineering, ops & security team members. The illustration below represents an ideal “Logging Lifecycle” and the steps including in each phase.
Get started now
To watch the full January webinar on-demand, click here.
If you haven’t already, create a free Logentries account to follow along during the webinar. Click here to get started.
