Last updated at Mon, 06 Nov 2017 18:45:46 GMT
Introduction
Web applications today are part of every IT operation within an organization.
Independent software vendors (ISV) as well as enterprises create web applications to support their customers, employees and even suppliers. All in all, the goal of these applications is to deliver a service, complete a transaction, support efficient and effective business operations, and directly impact company revenues and operational margins.
In 2009, Paypal went offline for over an hour due to a network infrastructure failure. At the time, Paypal was processing about $2,000 in payments every second, which meant its hour offline resulted in about $7.2 million in lost transactions. According to an Aberdeen research study, this type of downtime costs over $150K per hour, and the average recovery time is 5.13 hours. In severe cases such as Paypal’s, IT teams, which include operations and development, need to know what’s wrong—and fast. In these moments, a team’s ability to collect, analyze, and understand data in real-time is fundamental to resolving the cause of the problem, taking action and validating remediation. In this article, we define real-time analytics and demonstrate how log data from different layers of your systems and application stack can enable real-time analytics and response.
What is Real-Time Analytics?
When referring to “analytics,” people often think of manipulating an existing set of structured data to yield insights. “Real-time analytics” takes this definition a step further by accounting for the constant appending of new data to the existing data set and continuously re-analyzing the new dataset for new insights. But for analytics to be real-time, data needs to be ingested immediately upon creation, delivering results in a matter of seconds, enabling those interpreting the data to react right away.
To further demonstrate the definition of real-time analytics, let’s start by comparing it to the more commonly known, data batch processing. While batch processing can still append new data to an existing set, it does so in batches rather than a continuous stream. Batch processing comes with several disadvantages to real-time streaming. For example, if the data being processed doesn’t include timestamps, every event in a batch will be assigned the same timestamp (the date and time the batch process occurred). Batch processing also makes it impossibletogenerateimmediatealertsoffof events as they occur. In cases like Paypal’s, or any other web-based business, batch-process alerting is simply unacceptable.
In the world of operations and systems administration, log data is often seen as the ultimate source of real-time data, with systems often producing hundreds or thousands of log events per second. While a variety of tools exist for capturing log data, a tool’s ability to ingest and interpret log events in real-time as they occur is a key differentiator, with many tools taking anywhere from several seconds to several minutes to process a log event. A log management tool that isn’t truly real-time fails to capitalize on the true power that log analytics has to offer. How effective can a system alert be if you’ll still experience several minutes of downtime before even receiving the alert? Tools that are actually real-time can deliver information within seconds of occurring, alerting you to the warning signs leading up to an issue, improving your chances of identifying, diagnosing and resolving problems before they negatively impact end-users.
Four Real-Time Use Cases
Below, we explore four use cases that exemplify why real-time analytics are critical to performance and user experience, highlighting key capabilities that enable real-time analytics in each layer of your system or application:
1. The Application Layer
With your developer team preparing for a big push to production, you’re worried about the possibility of unforeseen issues immediately following the deployment. Testing in development will never provide an exact replica of what will happen in production. Therefore, the more you are able to view and monitor your logs in real-time, the faster you will be able to address and rectify issues. While big issues may be easy to spot, real-time analytics can also help you identify small issues building over time that could eventually slow down your application and user experience. While batch-processed analytics could only evergive you a historical analysis of your systems data, real-time analytics can enable you to identify anomalous patterns in your data as they occur. Using a log analytics tool that offers “anomaly alerts” can help you identify early warning signs of larger issues.
Regardless of the size of the release, whether it’s a minimal viable product released to a subsection of users or a large release following a three month sprint, things inevitably go wrong. Logs should be your go-to resource for investigating and addressing the source of issues that might arise while a real-time analytics tool should be used for alerting you to anomalous activity as it occurs.
“The easier it is to view and monitor your logs in real-time the faster to will be able to address and resolve issues.
2. The Database Layer
Imagine over the course of several minutes, your popular e-commerce application hasn’t received any orders. Where’s the first place you’d look for a possible issue? You may first check to see if your website is still reachable from a browser. Then, you may check your server logs. Or perhaps you check your APM tool? Or a web analytics tool? Are they all saying the same thing? Or nothing at all?When you notice there aren’t any errors in your code and traffic to your website appears to have remained steady, you decide to investigate your database. Only then, after wasting time investigating other scenarios, do you see your database was improperly configured in the last deployment and has reached its row limit. How many sales have you lost while guessing where to investigate?
Without log-based, real-time analytics, database errors can go undiscovered, often only realized after a period of noticeable inactivity and investigating. When using a real-time aggregated log analytics service, database errors stream into the same single view with the rest of your system’s log events as they occur. Alerts on database errors can be generated just as easily as alerts for the rest of your environment. And tools that offer custom tagging of specific event types can also help you spot database specific errors as they occur.
“How much revenue have you lost while guessing where to investigate the problem?
3. The Server/Hosting Layer
Let’s say your mobile app was just featured on Product Hunt and you’re suddenly experiencing a spike in traffic. Luckily, your app runs in an autoscaling environment and handles the load without issue. When the traffic later subsides and your servers scale back, you decide to analyze the distribution of 400 errors over time. But how will you access data from the servers that scaled down? If you weren’t sending those log files to a central location in real-time, your data is forever lost. In this scenario, centralizing your logs in real-time is crucial to capturing all relevant data.
When dealing with auto-scaling environments, real-time analytics also enables you to monitor the scaling processes in the moment, ensuring they scale as expected. Without real-time monitoring, you’d have no way of knowing if a server cluster fails to scale up. Likewise, if your environment fails to scale down when appropriate, you could be left paying for unused infrastructure.
“Real-time analytics enable you to monitor the scaling process, in the moment, to ensure everything is scaling as expected.
4. Cross System, Real-Time Analytics
Generating real-time alerts are an obvious necessity for reacting to issues as they occur. But what if your team isn’t seeing the alerts in real-time? Many tools only offer email notifications; but what if your team only checks emails a few times a day? Alerts are only as good as your team’s ability to react to them quickly. Therefore, it’s important to use services that easily and automatically integrate with the communication tools they’re already using, like Slack, HipChat or PagerDuty.
Cross-system, real-time analytics can also be demonstrated when integrating multiple monitoring tools. Consider New Relic, which offers real-time application performance monitoring: what if one of your application metrics recorded in New Relic requires deeper investigation? Manually switching from an APM to a logging tool to then search for related log events eats away at valuable time. In these cases, plugins that integrate APMs and log analytics tools for immediate log event correlation enables real-time action.
Final Notes
When it comes to leveraging analytics to ensure continuous service delivery and uptime, log-level data and real-time processing are two requirements for success. At every layer of your system’s hardware and application stack, real-time analytics enable centralized log collection and monitoring, easy identification of key events, and instant alerts to the communication tools used by your team. Thanks to the power of real-time analytics, you can improve your operational efficiency and prevent downtime, leading to a healthier, stronger business.
About Logentries
Logentries is the leading real-time log management and analytics service built for the cloud, making business insights from machine-generated log data easily accessible to development, IT and business operations teams of all sizes. With the broadest platform support and an open API, Logentries brings the value of log-level data to any system, to any team member, and to a community of more than 35,000 worldwide users. While traditional log management and analytics solutions require advanced technical skills to use and are costly to set-up, Logentries provides an alternative designed for managing huge amounts of data, visualizing insights that matter, and automating in-depth analytics and reporting across its global user community. To sign up for the free Logentries service, visit logentries.com
