Posts tagged Log Management

8 min AWS

Automating the Cloud: AWS Security Done Efficiently

Today, we are going to be installing software on all your existing EC2 instances across several (or all!) accounts under an organization in AWS.

2 min SIEM

SIEM Delivery Models: Where Do Today’s Risks and Future Technology Lead Us?

Recently, we partnered with Ultimate IT Security to discuss the current and future state of SIEM technology, and how it’s evolving to address current risks.

7 min Log Search

Rolling with Your Logs, Part 3: Using Regex to Expand Your Search Options

In this final installment of our Log Search series, we’ll look at some simple regular expressions that will greatly expand your Log Search options.

6 min Log Search

Rolling with Your Logs, Part 2: Advanced Mode Searches

In the Part 2 of this three-part series on InsightIDR Log Search, we will cover three concepts: parsed logs, groupby function, and log search operations.

5 min Log Search

Rolling with Your Logs, Part 1: Your Guide to Log Search in InsightIDR

In the first installment of this series, we'll cover the three most important basics of log search, then run through a few common Simple Mode searches.

6 min Log Management

Taking a Message-Based Approach to Logging

When you think about it, a log entry is really nothing more than a message that describes an event. As such, taking a message-based approach to logging by utilizing messaging technologies makes sense. Messaging creates the loose coupling that allows a logging system to be adaptable to the needs at hand and extensible over time. Understanding a Standard Logging Architecture Typically, logging is implemented in an application using a logger [https://docs.oracle.com/javase/7/docs/api/java/util/log

5 min IT Ops

6 Best Practices for Effective IT Troubleshooting

System monitoring and troubleshooting [https://www.rapid7.com/fundamentals/system-monitoring-and-troubleshooting/] can be a time-consuming and frustrating activity. It’s not unusual for IT folks to spend hours finding and fixing a problem that could have been resolved in 10 minutes had better troubleshooting tools and processes been in place. Improving IT troubleshooting and monitoring doesn’t need to be an expensive undertaking. Many times it’s just a matter of implementing a few company-wide

5 min Log Management

3 Steps to Building an Effective Log Management Policy

You’re on Call Duty. You’re awoken in the middle of the night by your cell phone in the throes of an SMS frenzy. You’re getting hundreds of messages from your company’s logging service: a record is being written to a database, code is being executed, a new container is being spun up, and on and on. None of these messages matter to you. You just turn off your phone and go back to sleep. The next day you go into the office only to find out that half the racks in your datacenter went offline durin

5 min InsightOps

5 Ways to Use Log Data to Analyze System Performance

Analyzing System Performance Using Log Data Recently we examined some of the most common behaviors that our community of 25,000 users looked for in their logs, with a particular focus on web server logs. In fact, our research identified the top 15 web server tags and alerts created by our customers—you can read more about these in our https://logentries.com/doc/community-insights/ section—and you can also easily create tags or alerts based on the patterns to identify these behaviors in your sys

4 min Log Management

What is Syslog?

This post has been written by Dr. Miao Wang, a Post-Doctoral Researcher at the Performance Engineering Lab at University College Dublin. This post is the first in a multi-part series of posts on the many options for collecting and forwarding log data from different platforms and the pros and cons of each. In this first post we will focus on Syslog, and will provide background on the Syslog protocol. What is Syslog? Syslog has been around for a number of decades and provides a protocol used for

3 min Log Management

Active vs. Passive Server Monitoring

Server monitoring [https://logentries.com/product/server-monitoring/] is a requirement, not a choice. It is used for your entire software stack, web-based enterprise suites, custom applications, e-commerce sites, local area networks, etc. Unmonitored servers are lost opportunities for optimization, difficult to maintain, more unpredictable, and more prone to failure. While it is very likely that your team has a log management and analysis [https://www.rapid7.com/products/insightops/] initiative

3 min Haxmas

12 Days of HaXmas: The Gift of Endpoint Visibility and Log Analytics

Merry HaXmas to you! Each year we mark the 12 Days of HaXmas [/tag/haxmas/] with 12 blog posts on hacking-related topics and roundups from the year. This year, we're highlighting some of the “gifts” we want to give back to the community. And while these gifts may not come wrapped with a bow, we hope you enjoy them. Machine generated log data is probably the simplest and one of the most used data source for everyday use cases such as troubleshooting, monitoring, security investigations … the lis

4 min InsightOps

Announcing InsightOps - Pioneering Endpoint Visibility and Log Analytics

Our mission at Rapid7 is to solve complex security and IT challenges with simple, innovative solutions. Late last year Logentries joined the Rapid7 family to help to drive this mission. The Logentries technology itself had been designed to reveal the power of log data to the world and had built a community of 50,000 users on the foundations of our real time, easy to use yet powerful log management and analytics engine. Today we are excited to announce InsightOps, the next generation of Logentri

6 min Incident Detection

User Behavior Analytics and Privacy: It's All About Respect

When I speak with prospects and customers about incident detection and response (IDR) [https://www.rapid7.com/solutions/incident-detection/], I'm almost always discussing the technical pros and cons. Companies look to Rapid7 to combine user behavior analytics (UBA) [https://www.rapid7.com/solutions/user-behavior-analytics/] with endpoint detection and log search to spot malicious behavior in their environment. It's an effective approach: an analytics engine that triggers based on known attack m

5 min Log Search

Log Search Simplified

Hi, I'm Laura, UX Designer at Logentries and today I'm going to discuss how just about anyone can use Logentries to search and analyze their log data no matter what their job title or technical skill level. What is Logentries? At Logentries, the team works tirelessly to provide an easy to use log management service [https://logentries.com/?CS=R7blog] that allows users to stream their logs from just about anything. Logentries can accept data from almost any device that generates log data, inclu

3 min Log Management

Using Log Data as Forensic Evidence

This is a guest post by Ed Tittel. Ed, a regular contributor to blog.logentries.com [https://blog.logentries.com/], has been writing about information security topics since the mid-1990s. He contributed to the first five editions of the CISSP Study Guide (Sybex, 6e, 2012, ISBN: 978-1-119-31427-3) and to two editions of Computer Forensics JumpStart (Sybex, 2e, 2011, ISBN: 978-0-470-93166-0), and still writes and blogs regularly on security topics for websites including Tom's IT Pro, GoCertify.co

3 min Log Management

Nexpose Logging Analytics using LogEntries

This blog shows how to use the power of LogEntries [https://logentries.com/] Search and Analytics to monitor your Nexpose installation. LogEntries has joined [/2015/10/13/why-we-re-welcoming-logentries-to-the-rapid7-family-a-story-of-data-and-analytics] the Rapid7 family and offers several powerful capabilities to search, analyze, monitor and alert on your Nexpose installation. LogEntries is also super easy to set up and maintain. I spent about five minutes getting it running. The Nexpose engi

5 min PCI

Seven Ways InsightIDR Helps Maintain PCI Compliance

If your company processes credit card transactions, you must be compliant with the Payment Card Industry Data Security Standard, or PCI DSS [https://www.pcisecuritystandards.org/documents/PCI_DSS-QRG-v3_2_1.pdf]. Any entity that stores, processes, or transmits cardholder data must abide by these requirements, which provide best practices for securing your cardholder data environment (CDE) [https://www.rapid7.com/solutions/compliance/pci-dss/]. Rapid7 InsightVM [https://www.rapid7.com/products/i

3 min Metasploit

Securing Your Metasploit Logs

Metasploit, backed by a community of 200,000 users and contributors is the most impactful penetration testing solution on the planet. With it, uncover weaknesses in your defenses, focus on the highest risks, and improve your security outcomes. Your Metasploit Pro console produces a lot of important logs. It is essential to be able to review these logs, alert on them, and keep them secure. Why should I monitor these logs? The logs produced by your Metasploit Pro console are helpful when troubl

5 min Log Management

If You Work In Operations, Your Security Team Needs The Logs, Too

This post is the final in a series examining the roles of search and analytics in the incident-detection-to-response lifecycle. To read the previous six, click one [/2015/10/21/search-will-always-be-a-part-of-incident-investigations], two [/2015/10/29/whether-or-not-siem-died-the-problems-remain], three [/2015/11/05/investigating-an-incident-doesnt-end-at-the-perimeter], four [/2015/11/11/making-sure-search-is-not-your-incident-response-bottleneck], five [/2015/11/19/siems-dont-detect-attacks-a

6 min Log Management

10 Best Practices for Log Management and Analytics

Introduction Today’s Log Management and Analytics Challenges Within the last decade, the advancement of distributed systems has introduced new complexities in managing log data. Today’s systems can include thousands of server instances or micro- service containers, each generating its own log data. With the rapid emergence and dominance of cloud-based systems, we have witnessed explosive growth in machine-generated log data. As a result, log management has become a staple in modern IT operatio

1 min Logentries

Logentries Joins the Rapid7 Family

I'm very excited today to join the Rapid7 family. The acquisition is good news for Logentries customers, Rapid7 customers and all of our employees.  It means that great minds and innovative technology have come together to solve some of our thorniest IT and security challenges. The Logentries team has been on a mission over the last few years -- Revealing the Power of Log Data to the World. While pursuing our mission, I am often asked why log data has become so valuable. The answer is simple: l

2 min Logentries

Why we're welcoming Logentries to the Rapid7 family - a story of data and analytics

Those that follow Rapid7 will know that we talk a great deal about our vision of delivering security data and analytics to our customers to enable an active, analytics-driven approach to cyber security. I'm excited to let you know that today we're making an important addition to the Rapid7 family that will help us advance this vision even further… we are acquiring the world-class, cloud-based log management and search technology company, Logentries. Organizations need real mastery of the inform

9 min Log Management

Q & A from the Incident Response & Investigation Webcast: "Storming the Breach, Part 1: Initial Infection Vector"

The recent webcast “Storming the Breach, Part 1: Initial Infection Vector [https://information.rapid7.com/storming-the-breach-part-1-initial-infection-vector.html?CS=blog] ”, with Incident Response experts Wade Woolwine [/author/wade-woolwine] and Mike Scutt sparked so many great questions from our live attendees that we didn't have time to get through all of them! Our presenters took the time to answer additional questions after the fact... so read on for the overflow Q&A on tips and tricks for

3 min IT Ops

Log Analysis for Orchestration Change Management

Are you suffering from server sprawl? You might be and don’t even know it. Server sprawl occurs when there are an unknown number of rogue VMs; VMs with unknown workload,  owners, or purpose. And no one is fearless enough to delete or suspend them. Orchestration tools make creating new nodes so easy that almost every organization who uses them is suffering from server sprawl…at a high price and lots of confusion. [/content/images/le-img/2015/04/log-analysis-for-orchestration-change-management-

5 min IT Ops

In the Log Management World: Are you a Fox or Hedgehog?

I’ve recently been reading Nate Silver’s book, “The Signal and the Noise.” In the book, Silver looks at a number of areas where predictions have been made and considers how successful they have been, as well as the reasons why they have been accurate (or not). I couldn’t help but draw the similarities how most companies use log management tools [https://logentries.com/doc/log-management/] today. Silver’s particular interests are political forecasting (see www.fivethirtyeight.com [http://www.

4 min IT Ops

The Pros and Cons of Open Source Logging

I recently attended DevOpsDays Boston [http://devopsdays.org/events/2014-boston/], which is a great way to get a pulse on the latest issues and technologies in the DevOps community. DevOpsDays is split between organized talks, open sessions and hanging around chatting with the other attendees. For me, the latter is always the most valuable. Over the 2 days, I probably spoke to more than 100 different people. I’m always interested in investigating how people are logging, what they are logging,

3 min IT Ops

Happy Bits Shortens Time-to-Resolution From 4 Days to 5 Minutes!

Happy Bits [http://www.happybits.co/] develops fun mobile apps so you can share videos instantly with your friends through messenger. You can either watch the videos live while you’re recording, or later whenever you have time. Here’s how it works: you upload a video to their app, then receive a private link to paste into your favorite messaging app to share the video. Happy Bits integrates with all the popular messenger apps, like Facebook, WhatsApp, Skype, Twitter, and Google+, just to name a

3 min IT Ops

Traditional Log Management Is Dead. Long Live Autonomic Analytics!

In this new era of cloud computing, faster and cheaper are not enough.  The modern Ops “toolkit” needs a log analytics service built for cloud-based environments that offers easy log data centralization, autonomic analysis, and a real-time monitoring service for connecting distributed systems and teams. The rapid emergence and dominance of cloud-based systems have contributed to an explosive growth of machine generated log data. We have heard from our community of more than 25,000 IT and Dev O

4 min IT Ops

Let the Logs do the Talking

You are on top of your game. You have a log analysis tool [http://logentries.com] churning logs from all your applications and infrastructure. And now that you have data (LOTS and lots of data…), you are able to understand your infrastructure better than you ever did before. You might even build a dashboard or two that tells you what is going on with your IOPS and utilization, at a glance. But when it comes time to explain system status or configurations to everyone else in the organization you

3 min IT Ops

Why we love log management (and you should too)

It doesn’t matter if your title puts you in Ops, Dev, or DevOps, being able to track down the big insights from your data is the secret-sauce every engineer is looking for. Collecting the data is a step in the right direction, but organizing, tagging, monitoring, and reporting from the data provides the insights necessary to make business decisions. Log data is a huge piece of that puzzle, and a good log management tool can really help your business run more efficiently. Our team loves log mana

4 min IT Ops

JSON logging in Apache and Nginx with Logentries

I’m often asked on calls with our customers what is the preferred format for log data being sent to Logentries [https://logentries.com/centralize-log-data-automatically/?le_trial=json_logging_in_apache_and_nginx-logentries_blog-post_cta-create_trial&utm_campaign=json_logging_in_apache_and_nginx&utm_source=logentries_blog&utm_medium=post_cta&utm_content=create_trial] . While we pride ourselves on being the Log Management [https://logentries.com/doc/log-management/] tool that is easiest to setup a

3 min IT Ops

The Log - So Important It Deserves a Book

I was very interested to see that Manning Publications have just announced the Early Access release of a new book called Unified Log Management [http://www.manning.com/dean/]. I looked through the table of contents and it was what I expected – a roll your own log management [https://logentries.com/doc/log-management/] cookbook with a list of technologies that might help (Kinesis, Redshift, Kafka etc.). Presumably it will also include Google DataFlow by the time the book ships. So it’s only reall

6 min IT Ops

5 Ways to Make Your DevOps Team More Efficient

As your DevOps team grows, scaling efficiencies across the group is imperative to maintaining a well-oiled unit. A small team of smart engineers can function well without much effort, but as your team gets bigger, you need to make sure you invest in the right tools and practices to help keep everyone on the same page. Throw in distributed teams, across different time zones, and issues can only get exasperated. Below are a few key tips that you should consider as your development and operations

2 min IT Ops

4 Ways to Use Centralized Log Management for Improved Production Monitoring

Is your team focused on preventing outages and minimizing downtime in production? Time to resolution is one of the most important operational KPIs for Ops teams, becauseany time that your application is down, is too long for your users. Whether it’s minutes, hours, or (in a worst case-scenario) days, any time that your systems are down, your business is losing money; and more importantly, customer trust and satisfaction. This creates additional pressure on your team and hurts employee morale. By

2 min IT Ops

Revisiting the Past with Logentries and Syslog

When I was younger I was always amazed by my grandfather. He would revel me in stories about how, when growing up, not only did he not have a car or television, but he had to walk up hill “both ways” to get to school – without shoes! And here I am today more or less saying the same thing about the late 70’s and early 80’s when client-server changeover that occurred during that time and we all had servers sitting in our closets. Setting up syslog [https://logentries.com/doc/about-syslog/] back in

2 min IT Ops

AWS CloudWatch Logs - Making Innovation Easier and Cheaper

Last week AWS made an exciting announcement at their NYC Summit, which I believe is a big step in helping to reduce the cost of log management in the cloud [https://logentries.com/doc/log-management/] for end users and vendors alike – good news all around! What was announced? Amazon announced ‘CloudWatch Logs’ which essentially allows you to send log data from your EC2 instances into CloudWatch for storage and some rudimentary analysis. How does it work? CloudWatch logs works by collecting y

5 min Log Management

Incident Response is about Where, When, and How

"If and when" is old and busted. "Where, when, and how" are the new hotness. Incidents happen. There will always be a Patient Zero. "Where the incident happened, when you detect the incident, and how you responded" is what I believe matters. I think the general public will appreciate measured response under attack to us fostering belief in 'perfect defense'. With this in mind, I want discuss a few thoughts prompted by eBay's response to this compromise. Scoping is hard Incident Handlers le

3 min PCI

PCI 30 seconds newsletter #19 - Your PCI Logbook - What is required in terms of log management?

P>D R is a well-known principle in security. It's a principle that means that the Protective measures in place must be strong enough to resist longer than the time required to Detect something wrong is happening and then React. For example, your door must be strong enough to prevent a malicious individual from getting in for at least the amount time required to detect the incident, alert the police, and have them arrive on site. In this context, log management plays a specific role. It help

4 min Log Management

Nexpose log files - What's changed in v5.2

Introduction Nexpose logs messages for tasks that the system has performed as well as events that occurred as a result of those tasks. The messages vary with respect to the features in the product such as users logging into Nexpose successfully, launching a scan for a site, or generating a report. The log files are helpful in understanding what Nexpose has already done. In the latest release, Nexpose 5.2, we have introduced a number of enhancements to the log files such as reducing disk usage an