How to Get Instant Insight into Your Log Data with Analytic Packs

Last updated at Mon, 27 Aug 2018 18:00:00 GMT

Useful log management only starts with centralizing your log data—the next step is making sense of that data so you can get valuable insights into your application and infrastructure performance.

InsightOps’ powerful log query language (LEQL) allows you to easily search your log data and run analytics for any number of metrics. Whether you want to see average memory usage over time, the number of HTTP 500 errors per page in your web application, or just search for all exceptions in your log data, LEQL has you covered.

However, what if you’re not too sure what you want to look for in your log data? This is where Analytic Packs can help. These are a set of pre-configured saved queries, alerts, and dashboard charts for a wide range of log data sources, including NGINX, Apache, IIS web access logs, syslog, and Windows Event logs. They take just a few minutes to add to your account and can help you immediately see the important trends and events in your log data.

So, let’s take a look at an example of Analytic Packs in action.

Installing an Analytic Pack

In this example, I’m collecting Docker metrics and events using the InsightOps Docker integration. InsightOps collects this data in JSON format, so it’s pretty readable. I can see statistics for CPU, memory, and network usage. It’s also collecting events, such as containers being started and stopped.

This is a good start—but this is really only raw log data so far. What are the important data points that I want to be alerted about or see in a dashboard? Let’s check out the Analytic Pack page and see how I can easily get this insight.

When I click on the Analytic Page link in the left-hand navigation, there’s a wide range of packs that I can add based on lots of different types of log data (there is a full list at the bottom of this blog).

To see the contents of a pack, I just click on the “Details” button. As we’re interested in seeing how I can drill into my Docker log data in more detail, I am going to click on the Docker pack.

Each pack has an overview of the pack’s contents, including a count of the dashboard charts, alerts, and saved queries that will be added. Here, I can see that the Docker pack contains 13 different dashboard charts, nine tags and alerts, and 21 saved queries.

Let’s go ahead and add the pack by clicking the “Add Pack” button. A panel will appear with a number of simple steps that will guide me through the process.

The first screen is the list of logs in my account that allows me to specify which logs I want the packs to be associated with. In this case, they are the logs being sent from the Docker logging container. As I progress through the steps, I will see a preview of the tags and alerts, saved queries, and dashboard charts that will be automatically added for me.

Using the new Analytic Pack in InsightOps

Great! The new pack has been added in just a couple of minutes, so what now? Let’s go back to the log search page and have a look at the Docker logs again.

When I click the “Queries” button, I can see that the saved queries from the pack have been successfully imported.

When I click one of the saved query names, it will run automatically.

Cool! I now have a breakdown of all the different container events in my log—and I didn’t even have to type a thing.

I’m also going to check out a breakdown of the Docker containers that were created.

So, what about the tags and alerts? When I scroll down to the raw log data below the results of my query, I can see those labels that were created earlier when the pack’s tags and alerts were imported.

As well as providing a visual cue for a notable event in my log data, labels can also be used for filtering so I only see the events I want to.

These labels in the UI are really useful for quickly filtering and visualizing my data, but I also may want to get notified when one of these events happens. When I click on the “Alerts” page link in the left-hand navigation, I can see the alerts that were added by my Analytic Pack.

From here, it’s just a case of editing the alert so it sends me an email. Now I can quickly react when an important event happens in my logs. Alternatively, I can have this alert sent to Slack, PagerDuty, Hipchat, Campfire, or a custom webhook URL, where you can hook it up to tools such as Komand to add powerful security automation and workflows that act on events from your log data.

Finally, I want to see an overview of all my data, so I’m going to have a look at the dashboard that was created for me.

Now I can see a high-level view of my log data. Important metrics are grouped by the image name. I can also see CPU and memory usage over time.

When I scroll down, I can see overall counts of important events. By clicking the settings icon, I can see the log data that powers these visualizations and then carry out any detailed investigation that is needed.

Summary

By adding an Analytic Pack, I was able to convert my raw log data into meaningful insight, set up alerts that will let me know as soon as something important happens, and see all this information in a dashboard—all in less than five minutes!

What packs are available?

Here is the list of packs that are available:

• Apache access logs: Combined Log Format and JSON
• Docker
• Heroku: Apps, log runtime metrics, and PostGres
• IIS Logs: IIS log format and JSON
• KeyCDN
• Linux Server Monitoring
• Microsoft DHCP
• Microsoft SQL Server
• MongoDB 3.0 monitoring
• NGINX Server monitoring: Combined Log Format and JSON
• Puppet: Console logs, database and master logs.
• Python APM
• Salesforce
• Syslog for Ubuntu
• Windows PCI Compliance
• Windows Security Events

Don’t see what you’re looking for? Let us know what log data you’d like to see a pack for!