Last updated at Mon, 20 Dec 2021 18:51:36 GMT
When you think about it, a log entry is really nothing more than a message that describes an event. As such, taking a message-based approach to logging by utilizing messaging technologies makes sense. Messaging creates the loose coupling that allows a logging system to be adaptable to the needs at hand and extensible over time.
Understanding a Standard Logging Architecture
Typically, logging is implemented in an application using a logger. A logger is an object or component that sends log entry data to a target destination. Binding a logger to one or many targets is done via logger configuration. The target can be a file, database, logging service, or depending on the logger technology, all of the above. Log4J, the most popular logger for Java programming, allows you to configure multiple log appenders to the logger, with each appender dedicated to a particular target. The same is true of Winston, a popular logger for Node.JS.
The important thing to understand in terms of application-based logging is that the application submits a log entry to a logger and then the logger forwards it onto one or more targets. How each target handles the log entry is its own concern.
Loggers are a reliable, well-known way to do logging. However, there is a drawback. Imagine that you have to add an additional target to a logger, for example a target that sends the log entry onto a mobile phone using SMS.
Adding the new target means having to deploy the appender as well as update and redeploy the logger configuration file, at the very least. You might have to take the entire application offline to deploy the new logging functionality.
Is there a way to avoid the problem? Yes, use a message-based approach to logging.
Taking a Message-Based Approach to Logging
As mentioned at the beginning of this article, a log entry is basically a message. In the case of using a target-based logger such as Log4J, the application sends the log entry to a logger and the logger has the wherewithal to format and send that log entry onto various targets. However, this scenario is limited in that the logger needs to know at the onset the various targets in play. Adding targets after the logger is deployed is difficult.
By taking a message-based approach, you can avoid having to fidget with an application’s logger to add new targets. In fact, the notion of “target” goes away altogether and is replaced by the concept of a subscriber.
Figure 2 shows a simple message-based logging architecture using the pub-sub pattern.
In a message-based approach, the logger emits a log entry as a message to a single target, a Message Broker. A Message Broker publishes an entry point, usually an HTTP URL that accepts messages as a POST request. Conceptually that endpoint is an exchange, a place where messages are collected and then distributed to interested parties. The term used by Amazon Simple Notification Services for an exchange is a topic. RabbitMQ, another popular messaging technology, uses the term exchange.
The benefit of using an exchange is that one or more subscribers can bind to it and receive a copy of any message sent. This is the basis of the publish-subscribe pattern or pub-sub, for short. Using pub-sub architecture means that subscribers that come online later can bind to the exchange, a.k.a publisher, and get messages that it can use for its own purpose.
Thus, going back to the late binding SMS feature scenario described earlier, all that is required to support accepting log entries and forwarding them on as SMS is to create a subscriber that knows how to convert log messages into an SMS format and then send them on. After the subscriber is created, it gets bound to the publisher. Once bound, the subscriber received a copy of the message sent to the publisher. Most publisher technologies have the ability to keep sending the message to subscribers a set number of times until it is accepted. Undeliverable messages get noted by the publisher.
Should it be determined at a later time that the SMS subscriber is no longer needed, decommissioning the subscriber requires nothing more than disconnecting it from the publisher. No code in the publisher needs to be adjusted. Logging continues from the application unimpeded.
The nice thing about a message-driven logging architecture is that it provides a great deal of flexibility. You can have a subscriber that forwards messages onto a standard logging service, such as InsightOps, to database services such as AWS DynamoDB, or to a file out on Azure. Your application needs to know nothing about the eventual targets. All the application knows is that it’s sending log entries to a single target, the publisher.
Does Message-Based Logging Make Sense for Your Company?
Implementing a message-based approach to logging is not for everyone. First, if your company does not already have a messaging architecture in place, it needs to have the expertise and capacity to implement one. You can use a cloud service such as AWS SNS/SQS, Azure Service Bus, or Google Pub-Sub. Also, you can take a server-based approach using industry standard messaging products such as RabbitMQ or Apache’s Kafka. Regardless of the approach you take, your company will need to devote attention and resources to implementing and supporting a messaging framework.
Secondly, all of the physical messages sent to the publisher need to adhere to a conventional format. As the name implies, core to implementing a message-based architecture is understanding that all emitted log data takes the form of a message. Any message can end up anywhere. Thus, you don’t want to be sending around messages that require a lot of predefined knowledge to decipher. You need to have a message format that is conventional, self-describing, and extensible.
Listing 1 below shows an example of a message that is not self-describing.
"My Cool App","ip: "10.1.241.116","log_event","INFO","Saving data"," {"name": \"Bob\", \"status\": \"cool\"}"
The semantics of the message shown above are unknown. You need an outside reference to decipher it. As a result, writing the parsing algorithm to make sense of the message becomes a time consuming undertaking, specific to that given message format. Rather than using a custom message format, it’s better to use a conventional, self-describing format such as JSON, XML, or YAML. Self-describing messages are easy to understand and easy to parse. Using a conventional, self-describing format avoids the hazard of needing “tribal knowledge” to figure out the semantics of a log entry when designing a subscriber, particularly when the subscriber consuming the message binds to the publisher months after the publisher has been put in play.
Listing 2 shows a message that is self-describing. It’s in JSON format and thus can be understood clearly by convention.
{
"source": "My Cool App",
"ip": "10.1.241.116",
"time": "Fri Feb 28 07:25:25 UTC 2016",
"type": "log_event",
"body": {
"level": "INFO",
"message": "Saving data",
"data": {"name": "Bob", "status": "cool"}
}
}
Organizations that have a history of working with message-based architectures understand the value of constructing message using conventional formats. You want to have an environment in which any message can be consumed at any time, by any subscriber, in a manner that is standardized and accurate. Companies are not born with such capabilities. They need to develop them. If your company has the appetite for such an undertaking, you’ll do well. If not, it might be best to stick with the current approach to logging.
Putting It All Together
Adopting a message-based approach to logging has definite benefits. You can provide log information to any interested consumer at any time during an application's lifetime. Also, when a given consumer no longer has need of the log data, it can easily disconnect from the log message stream.
However, using a message-based approach to logging requires that enterprises have the appetite and wherewithal to support a messaging architecture. However, having a messaging architecture in place is not enough. Once a messaging architecture is adopted, the messages that travel from publisher to consumer should adhere to a conventional format that is easy to parse and process.
Message-based logging provides a great deal of flexibility. Such flexibility is particularly useful in dynamic organizations that need to adapt quickly to continually changing logging demands. Implementing a message-based approach to logging might require a company to make some fundamental changes in the way its system does logging. However, the long term benefits that the messaging brings make the effort well worth the investment.
