Last updated at Thu, 12 Dec 2019 16:31:35 GMT
This blog post is the second and final in our “Be Audit You Can Be” series. Be sure to check out part one, which covers how to securely send and monitor your audit logs with InsightIDR.
I could look at logs all day long. If you are reading this article, you probably feel the same way! There are so many hidden gems nested in logs, although admittedly sometimes they are hard to find, especially when the logs are not normalized as you want.
Let’s take a look at how InsightIDR’s Custom Data Parsing tool can make quick work of parsing out those interesting fields in the logs. I am using as my example logs both the audit and auth logs from InsightVM, our vulnerability management tool , but you can bring in a lot of different types of logs into InsightIDR, our SIEM security tool.
First, you do need to collect the logs and get them into InsightIDR.
For the audit.log and auth.log, I have followed the steps in this detailed blog post.
Let’s begin by finding the logs in Log Search. You can use the Custom Data Parser to parse out unparsed fields of data from already parsed logs, but in this article I will discuss using it for completely unparsed logs. I don’t want to call anyone’s baby ugly, but as you know, some of those unparsed logs can be messy and just downright difficult! The Custom Data Parser is going to help us with that.
In Log Search, find the option to Extract Fields, which is the top right-hand corner of the page.
Use the wizard to give the Parsing Rule a name and select the event source to be parsed. In the Extract Field wizard, you will use the first event that shows up in the list as your example log for the extraction process. If needed, use the Filter box to filter the types of events to just one type. If you do use the Filter, you will likely need to create more than one Parsing Rule. For example, the audit.log has two types of events in it: changes to users and changes to sites. These two types of events have different fields in them, and consequently, I need to create two Parsing Rules.
With the filter in place if you are using one, highlight the first value that should be parsed out. The Data Parsing Tool will highlight in yellow what data will be parsed out in the rest of the logs in your sample, so you can verify that are getting the data you need pulled out for this field.
Once you have the data properly highlighted, enter a name for the field and click the Add Field button.
This is probably a good time to stop and think about how you want to name the fields. Especially if you think you will be creating a lot of custom parsing rules, creating a convention for yourself now will make things easier later. I decided that for my organization, I will always use lowercase for the field names, and I will always separate words with underscores, for example site_name.
Keep adding fields until you have pulled out all the data that you want. When you have finished, select Apply Rules. Next, you will be asked if this Parsing Rule should be applied to any other event sources. Check any that apply, select Save & Finish, and continue with the wizard to save and complete your Parsing Rule.
Once you have created all the rules that you need, you should make sure that the Parsing Rules work as you intended. The rules will not be applied to logs that have already been collected, so after some more logs have been collected, use Log Search to verify that the fields you need are being parsed out.
Your custom fields are parsed out as nested JSON, so do be sure to specify both the parent and child fields when using them.
You can find more information on how to query logs in nested JSON format here.
My video shows how to parse out both the audit.log and auth.log.
Now that I have parsed the audit.log, it is no longer ugly and difficult. I can easily search the logs, create Custom Alerts on the parsed fields, or use them in my Dashboards. I was able to do all of this in a few minutes, and best of all, I never had to use any dreaded regular expression! I hope you will stop avoiding your ugly logs and give a try at parsing them with the Custom Data Parser, too.
