Last updated at Mon, 06 Nov 2017 18:27:34 GMT

Once you learn what the normal baseline on your network is, you should start to look for abnormal activities. This will help you understand any sudden changes that might affect the overall performance and help in easier troubleshooting. If you are already caching with Fastly, you might seek to pull more information from your log data to give you deeper visibility. There are a couple of formats in which you can send logs from Fastly to Logentries. You get a choice of either using Fastly’s default logging format, which is essentially an Apache Log Format, or you can jump into custom VCL (Varnish Configuration Language) and use the available VCL extensions to add more details to your log events. In this article, we take the second approach and present techniques that might be used by Fastly users in conjunction with the Logentries real-time log management analytics service.

Today, we are announcing the release of our new Fastly Community Pack

equipped with a number of powerful “plug and play” Tags, Saved Queries and Visualizations to help analyze your Fastly log data. The Community Pack can be easily imported into your Logentries account so that you can get quick visibility into your Fastly logs.

What’s included in the Fastly Community Pack?

Searches:

  • Device Detection
  • Unique IP Count
  • Requests by: country, region, Fastly datacenter, country code, country name
  • Status Codes
  • Total Hits over time
  • Average Hits over time
  • Total Data downloaded

Tags & Alerts:

  • 200-OK tag
  • 404-NOT FOUND alert
  • 20% increase in a download of files larger than 1MB alert
  • Application Not Accessed for the past 24h alert
  • iPad user tag
  • iPhone user tag
  • Android user tag
  • Desktop user tag

Graphs:

  • Device Detection pie chart
  • Unique IP counter
  • Requests by Country
  • Requests by Fastly Datacenter
  • Status Codes bar chart
  • Hits over time line chart
  • Average Request Size line chart
  • Total Data Downloaded

Fastly logs are fitted out with many interesting caching related variables by default. In addition to the above, the Pack allows you to directly configure caching using VCL extensions. VCL together with a powerful log analysis  platform enables you to view data from different angles and makes log analytics more dynamic and valuable to monitoring your overall infrastructure. To get a better understanding of the process we will present a couple of interesting approaches you can introduce to the standard VCL file.

Simple Device Detection Mechanism

First, use one of the VCL extensions variable req.http.User-Agent in order to get started and throw in few simple if-statements at the top of the main VCL file.

#FASTLY recv

    if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
      return(pass);
    }
    
if (req.http.User-Agent ~ "iPad") {set req.http.X-Device = "iPad";}
else if (req.http.User-Agent ~ "iPhone" ) {set req.http.X-Device = "iPhone";}
else if (req.http.User-Agent ~ "Android" ) {set req.http.X-Device = "Android";}
else {set req.http.X-Device = "desktop";}

    return(lookup);
}

Whichever type of machine or browser used to access your cached application via Fastly gets picked up by req.http.User-Agent. The above code example is simply a customization and shortening of already existing user agent standard format to make the log output more readable and to allow for easy identification of different device types. For example, you might be interested in a creation of a set of tags to identify distinct mobile users from desktop users. Using the above code, you can store the type of a device in req.http.X-Device variable which can then be used in your log output as follows:

log {"syslog LE :: client_ip="} req.http.Fastly-Client-IP {" devicetype="} req.http.X-Device;

Let’s note that {” devicetype=”} is a label for the *req.http.X-Device *– this allow you to send log entries in a form of Key Value Pairs (KVP), which broadens your search capabilities. It might be helpful to take a look at how to send logs from Fastly account to Logentries dashboard.

The logs are sent to Logentries, what’s next?

Once you have everything set up and are successfully feeding Logentries with Fastly logs, it is time to make use of the data and perform some interesting searches.  First, you are able to create a pie chart to outline the breakdown of devices used to access your cached application. This and many more searches are included in the free Community Pack.

You can customise your VCL to detect what type of machine accessed your application cached on Fastly.com

What about real-time tagging?

You are able to create custom tags using Logentries or simply import it straight from Fastly Community Pack (Please note that variable names in your custom VCL file need to be matching with variable names defined in JSON Community Pack File downloaded)

Custom tags are used to mark important events or actions

GeoIP Related Logs

One of the most exciting features of a VCL is the ability to feed your logs with approximate geographical location such as:

  • Latitude, Longitude
  • City
  • Continent Code
  • Country Code
  • Country Name
  • Postal Code
  • Region
  • Area Code
  • Metro Code
  • Server-Region
  • Fastly Datacenter

You might have already seen a good example of how GeoIP variables can be used in conjunction with the custom VCL. In the new Logentries’ Fastly Community Pack you can take it a step further and use the above variables to create some useful graphs based on your searches.

With current growth rate, Fastly is bringing online 1.5 POPs a month, and this will only increase as time goes on.

You can gain this data by adding your VCL extension to the main VCL file

User data like this has a great value and potential. The more you search, plot or analyze, the more you learn about your users.

Size Related Visualization

For improved analytics and more insight into what happened in a request, Fastly added size-related variables to VCL, such as:

  • Total size of a request
  • Total size of a header
  • Total size of a body
  • Total size of a response
  • Size of a header in a response
  • Size of a body in a response
  • Whether the response was successfully completed or not

Total data downloaded over a period of time in bytes.

You can add the variables above to your logs and use it to create Anomaly Alerts or graphs to visualize your network load and request characteristics.

Never enough tools!

Getting to know  your users is crucial, but knowing your data is equally important. This is why the Fastly Community Pack was equipped with searches and graphs such as Unique IP counter, size related searches and requests information analytics. We encourage you to explore our sample anomaly and inactivity alerts included in this Pack to power up your first line of defense in application monitoring. The main reason why you would create alerts like this is to ensure that you always have up-to-the-second data about normal and abnormal activities on your network, enabling you to react more quickly and solve the issue before your users notice.