Last updated at Mon, 06 Nov 2017 18:44:26 GMT

Over the last year we have watched Docker take the world by storm and when you stop to ask “Why?”, it becomes quite obvious: Docker may eventually replace the hypervisor (see diagram below) with a more lightweight efficient and scalable approach to building tomorrow’s systems. That’s a pretty big deal, so no wonder we are seeing so many organizations moving in this direction and a diverse ecosystem springing up around Docker.

At Logentries we have been witnessing this movement first hand and have noticed the migration of customers and users from early adopters “playing” with Docker environments to Ops teams running very large workloads on them. However, with real workloads comes real responsibility; and to date, monitoring these environments can have its challenges. Initially, simply getting the relevant monitoring information from your container environment and figuring out the right way to collect this had people scratching their heads.

But recent additions to the Docker platform have been a big help, in particular:

  • The stats API: Docker 1.5, introduced a new stats API endpoint and CLI command that will stream live CPU, memory, network IO and block IO for your containers. The API endpoint can be used to build tools that feed live resource information for your containers into your existing monitoring solutions, or you can build live dashboards directly using the API.

  • Docker logging drivers:Docker 1.6 introduced the new logging drivers which allow you to send container logs to other systems such as a remote syslog server or a third-party logging provider. Docker 1.7 has added support for a journal driver and Docker 1.8 is expected to bring driver support for Fluentd and Gelf.

However, getting a comprehensive view across your entire Docker stack has historically involved using multiple tools to do so.

So, what is comprehensive Docker monitoring?

Comprehensive Docker monitoring includes collecting log data and stats from the different layers across your container environment, with the easy ability to monitor and correlate them in a single place.

Looking at the container stack shown above, the following information is incredibly useful to give you a comprehensive monitoring solution:

  • Per container Logs: Logs produced by whatever is running inside your container, which may container APM data, system usage patterns as well as any system errors.
  • Per container Stats: Per container stats give you visibility into resources (CPU, Network, Mem etc.) being consumed by each container.
  • Docker API Events: Docker API events capture container lifecycle activity within your environment and can be used to give visibility into when containers are spun up and down; or if specific containers are killed and when. Think of this data as an audit trail of your container lifecycle.
  • Application Performance Metrics: This data is key for understanding service latencies and opportunities for code optimizations.  Application performance monitoring data is more and more often  included in log data, as a lightweight solution for understanding code performance.
  • Host Logs: Logs from the host OS such as anything from /var/log/ as well as the Docker daemon log will give you lower level insight into what is happening at the host level.
  • Host Stats: Resource usage data from the host will help you get a top-to-bottom view of performance issues or situations where resource constraints are leading to scalability issues. This is especially useful when viewed alongside container stats info.

Today, Logentries announced some cool updates to our logging container that allow you to easily get per container logs and stats, APM data, as well as Docker API events from your Docker environment. If you also want the host level information, you can also easily collect host-level resource usage information and log data using the Logentries agent to give you visibility across the full container stack – providing you with a single place to view, correlate and analyze all this data.

Here’s what our friends and Docker experts at Nearform say about it:

“As the technology landscape changes, some things remain constant. Consistent and accurate logging is one of those constants, required in all applications. We implemented the Logentries Docker container to make log aggregation and analysis in a containerized environment a no-brainer to set up and operate. This latest update adds the ability to record container events in addition to application logs and container statistics to provide increased visibility into system behaviour”

As always let us know what YOU think!