Technical Resources
Educational Resources
APM Integrated Experience
Connect with Us
Writing logs in a machine-readable format ensures the log can be leveraged by software.
First, to learn what structured logging is, you must take a step back and understand what exactly is unstructured logging.
With unstructured logging, events are expressed in plain text—in other words, plain English (or another language). The assumption here is that humans are the main target audience for logs, which isn’t always the case.
For instance, being able to search through your log files to find all occurrences of a given event is valuable if you’re trying to troubleshoot some issue or investigating a concerning trend.
Structured logging makes this easier by generating logs in more easily parsable formats—such as JSON and XML. This way, you can treat your log events as data rather than mere text.
Structured logging addresses the limitations of regular logging, bringing benefits for different roles in the organization:
Structured logging acknowledges that machines are the most common readers of logs nowadays, while also striking a nice balance with the need for human readability.
Say your application lets users create reports, which you want to log. The bare minimum log message for that could look like this:
2021-07-29 | New report createdYou have the date in ISO-8601 format, followed by the message. It could use some more context, such as the time of day and a log level to express the severity of the event:
2021-07-29 14:52:55.1623|INFO|New report createdThings are getting better. Let’s also include the ID of the user who created the report:
2021-07-29 14:54:55.1623|INFO|New report created by user 4253It wouldn’t be a stretch to imagine log events recording different occurrences:
INFO|New report created by user 4253
INFO|Report updated by user 257
INFO|User 478 deleted report
As you can see, users can also change and delete ports. Also, the last example doesn’t use the same format as the other two.
Now comes the tricky part. Let’s say you need to search through your log events to find all instances of a given user performing some action to reports. That’s not so easy to do, thanks to the inconsistent format of the messages.
Besides reports, users can create other types of artifacts, and such activities are also logged. With the right amount of regex, you’ll probably be able to pull it off. However, such code is error-prone, fragile, and doesn’t generate the most value for you and your customers.
Structured logging solves the above problems by generating easily parsable logs. For instance, the following is an example of how we could structure the events in the example using JSON:
{
"TimeStamp": "2021-07-29 14:52:55.1623",
"Level": "Info",
"Message": "New report created",
"UserId": "4253",
"ReportId": "4567",
"Action": "Report_Creation"
}
We have properties that indicate what action occurred and the identifications of the relevant objects, such as the user and the report created. Parsing a collection of events in the format above makes it easier to search through events than it would be with plain text.
You can use structured logging in any major language. However, we have to pick a language for the tutorial, so we’ll go with C#. And despite there being alternatives, we’ll pick Serilog for our logging framework.
Before we start, let’s make sure you have everything you need:
For brevity’s sake, I won’t walk you through creating a .NET app. Instead, there’s a sample app you can get from this GitHub repo. Clone the repo using Git or download the whole application as a .zip file.
After cloning or downloading the application, access the extracted/cloned folder and open it using your favorite editor. You should see the following structure:
We have properties that indicate what action occurred and the identifications of the relevant objects, such as the user and the report created. Parsing a collection of events in the format above makes it easier to search through events than it would be with plain text.
You can use structured logging in any major language. However, we have to pick a language for the tutorial, so we’ll go with C#. And despite there being alternatives, we’ll pick Serilog for our logging framework.
Before we start, let’s make sure you have everything you need:
For brevity’s sake, I won’t walk you through creating a .NET app. Instead, there’s a sample app you can get from this GitHub repo. Clone the repo using Git or download the whole application as a .zip file.
After cloning or downloading the application, access the extracted/cloned folder and open it using your favorite editor. You should see the following structure:
Inside the src folder, you’ll find the SampleApp folder and, inside it, the Program.cs class. Open it. Here’s what it looks like:
It’s a very simple app: it prompts you to type a username. If the username is a valid GitHub user, it then displays the list of public repositories for that user.
The next step is to add Serilog to our app. We’ll do that using Nuget.
Open up your terminal and navigate to the root folder of the project. Then, run the following commands:
cd src/SampleApp
dotnet add package Serilog --version 2.10.0
dotnet add package Serilog.Sinks.File --version 5.0.0
dotnet add package Serilog.Formatting.Compact --version 1.1.0
With the commands above, you navigate to the project’s folder and install the necessary packages we’ll need to work with Serilog.
Serilog has the concept of sinks. These components effectively write your log message to its destination. The list of available sinks for Serilog is long. For our tutorial, we’re using the file sink.
First, start by adding the necessary using statements to the top of your class:
using Serilog;
using Serilog.Formatting.Compact;
Subsequently, configure your log. Right at the top of the Main method, add the following lines of code:
Log.Logger = new LoggerConfiguration()
.WriteTo.File(new CompactJsonFormatter(), "log.json")
.CreateLogger();
With the lines above, you create a new logger that uses a file sink, which will write to a file called “log.json” using a JSON formatter.
You’re now ready to log something using the structured approach. In the code, go immediately after this line:
WriteLine($"The user {user.Name} was successfully retrieved!");Then, paste the following code:
var result = new {
Action = "USER_RETRIEVAL",
Success = true,
Fullname = user.Name,
Login = username};
Log.Information("{@Result}", result);
In the lines above, we create a variable using C#’s anonymous types feature. The object represents the fact that we’ve successfully retrieved a GitHub user.
Then, we use the logger to log the object with the Information level.
Now you can simply run the app and type a valid GitHub username when prompted. Then, navigate to the output folder of the project and you’ll find a log.json file there. Open it and you’ll see something like this:
{
"@t": "2021-07-30T00:08:03.2619130Z",
"@mt": "{@Result}",
"Result": {
"Action": "USER_RETRIEVAL",
"Success": true,
"Fullname": "Phil Haack",
"Login": "haacked"
}
}
(For my tests, I’ve used Phil Haack’s username.)
We get a nice JSON containing the properties of the result object we’ve created and the timestamp for the event. A notable absence is the log level. However, this is by design: omission of level implies INFORMATION.
For the next step, we’ll add a log to the part of the code that handles a user not found.
Right after this line:
WriteLine($"{username} isn't a valid GitHub user!");Paste the following code:
var result = new {
Action = "USER_RETRIEVAL",
Success = false,
Login = username};
Log.Error("{@Result}", result);
The code is virtually the same code we’ve used before, with only two differences:
To test, you only have to run the application and, when prompted, type an invalid username.
After that, go back to the log.json file, and you’ll find a new entry:
{
"@t": "2021-07-30T00:18:18.4842413Z",
"@mt": "{@Result}",
"@l": "Error",
"Result": {
"Action": "USER_RETRIEVAL",
"Success": false,
"Login": "itsharderthaniexpected"
}
}
Now, the resulting JSON contains a property depicting the log level.
No matter how great your QA strategy is, things will go wrong in production. That’s why logging is so valuable: it’s an essential part of troubleshooting applications.
What should your next steps be? For starters, continue to learn about structured logging, such as message templates.
Then, maybe you’ll want to start looking at tools to take your logging approach to the next level, such as log management/log analysis tools.
SolarWinds® Loggly® is a cloud-based log management tool you can use to aggregate and analyze your logs from virtually any source. To make things better, Serilog’s long list of sinks include a sink to Loggly. We invite you to create your Loggly account today.
This post was written by Carlos Schults. Carlos is a consultant and software engineer with experience in desktop, web, and mobile development. Though his primary language is C#, he has experience with a number of languages and platforms. His main interests include automated testing, version control, and code quality.
