Syslog-ng Configuration and Troubleshooting Tips

Syslog-ng is a system logging application. It provides logical separation between applications and their log messages, so they can simply “fire and forget” log messages to a centralized process for reporting, analysis, and storage. In this post, you’ll learn the basics of syslog-ng configuration. We’ll cover how to install, configure, and test syslog-ng on your Linux system.

After you configure and test a basic setup, you’ll add a new logging destination and test it.

Syslog-ng Overview

Syslog-ng messages can be stored to local disk, forwarded to another login daemon, or both. Each log message has a code indicating the software type generating the message, a severity level, a timestamp and the actual message. In addition to getting messages to where they need to go, syslog-ng also takes care of housekeeping such as making sure log files don’t get too big and archiving or deleting old files as new ones are created.

Syslog-ng is an implementation of the syslog protocol commonly found on Unix and Unix-like systems, with several important additions such as:

• TCP transport for sending messages to remote destinations
• Message queue support for distributing messages, including Kafka, AMQP, and STOMP
• TLS support for encrypting messages on the network
• Logging messages directly to SQL and noSQL databases
• Wild carding in messages and in the files used to store them
• Flow control for messages sent across the network
• Several options for formatting and rewriting log messages using built-in parsers
• Native support for sending logs to big data clusters and third-party log handlers including SolarWinds^® Loggly^® and Elasticsearch

Syslog-ng is available both as an open-source product and a commercial product. We’ll cover the open-source edition in this tutorial.

Installing Syslog-ng

You can download install packages for all the major Linux, Unix, and MacOS releases here. Unfortunately, the only official releases for Windows are distributed as part of the commercial product. The same syslog-ng daemon acts as a client, server, or both, so you don’t need to install different packages.

Follow the instructions for your operating system to install the syslog-ng daemon. Once it’s installed, you’re ready to move on to configuring syslog-ng.

Syslog-ng Configuration

Configuring syslog-ng is simple. Find and edit the syslog-ng.conf file. On most distributions you’ll find it in the /etc/syslog-ng/ directory. You can edit the file with your favorite text editor.

The config file syntax is specific to syslog-ng but should look familiar to most programmers. We’ll go over the file structure as we review a few different configuration options.

What Do You Want To Do?

As discussed above, the capabilities of syslog-ng are far too extensive to go through in a single blog post. So, let’s pick a few simple ones to illustrate how to configure this powerful logging tool. Let’s start by setting up a syslog-ng daemon to log to a file. Then, we’ll set up logging to email. Finally, we’ll briefly cover how to send messages to SolarWinds Loggly. Your syslog-ng package should have come with a basic configuration file that already performs basic system logging.

Version

The first line of your configuration must have a version declaration. This is required. Your version may be different, but we’ll be covering features from version 3.13 and up.

@version: 3.13

Source

A few lines down, you’ll see a source declaration like this:

source s_src {

       system();

       internal();

};

These four lines make up a single declaration.

1. All declarations begin with a type. This entry defines a source. Configuration files may have more than one source.
2. Declarations have a name as the second entry. This source is named s_src.
3. Next come the definition parameters contained between the two curly braces.
   1. system() is a special definition encompassing system syslog entries.
   2. internal() is another special definition with syslog-ng internal log messages.
4. A declaration ends with a semicolon.

This declaration creates a source named s_src comprising system and syslog-ng internal messages.

Now we need to send those messages somewhere.

Destination

We’ve created an input into syslog-ng. Now we need to set up outputs. We want more than one output because system() is a firehose containing all the syslog messages created on the host. If we put them all in one file, it’ll grow too large too quickly and be very difficult to read.

Here’s an excerpt from the sample configuration file.

destination d_auth { file("/var/log/auth.log"); };

destination d_cron { file("/var/log/cron.log"); };

destination d_daemon { file("/var/log/daemon.log"); };

destination d_kern { file("/var/log/kern.log"); };

destination d_lpr { file("/var/log/lpr.log"); };

destination d_mail { file("/var/log/mail.log"); };

destination d_syslog { file("/var/log/syslog"); };

destination d_user { file("/var/log/user.log"); };

destination d_uucp { file("/var/log/uucp.log"); };

destination d_debug { file("/var/log/debug"); };

The declarations look very similar to the source declaration above. Each one starts with the destination type. Next there’s a name, and then the parameters, which are all files() with a full path to where you want syslog-ng to put the file.

How do the messages get placed into these files?

Filter

Why do you have one source and many files? Our source contains all the input messages to syslog-ng. If you put them all in one file, it’s going to be almost worthless. You need to use filters to organize the messages.

filter f_dbg { level(debug); };

filter f_info { level(info); };

filter f_notice { level(notice); };

filter f_warn { level(warn); };

filter f_err { level(err); };

filter f_crit { level(crit .. emerg); };




filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };

filter f_error { level(err .. emerg) ; };

filter f_messages { level(info,notice,warn) and

                    not facility(auth,authpriv,cron,daemon,mail,news); };




filter f_auth { facility(auth, authpriv) and not filter(f_debug); };

filter f_cron { facility(cron) and not filter(f_debug); };

filter f_daemon { facility(daemon) and not filter(f_debug); };

filter f_kern { facility(kern) and not filter(f_debug); };

filter f_lpr { facility(lpr) and not filter(f_debug); };

filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); };

filter f_mail { facility(mail) and not filter(f_debug); };

filter f_news { facility(news) and not filter(f_debug); };

filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); };

filter f_user { facility(user) and not filter(f_debug); };

By now the configuration entries should look familiar. Each filter has a name and parameters. The parameters contain level() and facility() functions. Syslog-ng has a large set of functions you can use in your filters.

Here are a few common filters:

• facility()matches messages using the facility. We’ll look at this when we test the configuration.
• level() or priority()matches messages based on their priority.
• filter()matches based on a previously defined filter. The filter f_user will only match messages not already filtered by f_debug.
• host()matches based on the sending host.
• match()matches with a regular expression based on a specified header or content field.
• message()matches with a regular expression based on their message content.
• program()matches using the sending application.
• source()matches based on the source.

Log Paths

Now it’s time to tie everything together.

log { source(s_src); filter(f_auth); destination(d_auth); };

log { source(s_src); filter(f_cron); destination(d_cron); };

log { source(s_src); filter(f_daemon); destination(d_daemon); };

log { source(s_src); filter(f_kern); destination(d_kern); };

log { source(s_src); filter(f_lpr); destination(d_lpr); };

log { source(s_src); filter(f_syslog3); destination(d_syslog); };

log { source(s_src); filter(f_user); destination(d_user); };

log { source(s_src); filter(f_debug); destination(d_debug); };

A log combines a source, filter, and destination.

Let’s test out a configuration.

Testing Syslog-ng Configuration

You’re going to use the logger utility to verify your syslog-ng configuration. This tool is probably already installed on your system. If not, check your distribution’s documentation for instructions on how to add it.

You specify a message facility and priority with the -p option. Let’s send a message with the daemon facility and debug priority.

$ logger -p daemon.debug "This is a test."

Where will this message end up?

• It will arrive via the s_src source, since we’re piping all messages through there.
• It will match the f_debug filter, since the level is debug and the facility is
• It will match the last log entry above.

Let’s check in /var/log/debug

You’ll see a message with your login name and the test log message.

Sep  7 11:32:40 hala egoebelbecker[4207]: This is a test.

Alternate Destinations

Now you’ll add a new destination to your syslog-ng configuration and test it.

SMTP

smtp() is the mail destination for syslog-ng. With it, you can send messages to email instead of or in addition to files. Be careful with this one, though. It will generate an email for every log message match, which means you should always be highly selective here.

First you need the destination:

destination d_smtp {

    smtp(

        host("localhost")

        port(25)

        from("syslog-ng alert service" "noreply@example.com")

        to("Admin #1" "egoebelbecker@localhost")

        subject("[DEBUG] Debug log message of $LEVEL condition received from $HOST/$PROGRAM!")

        body("Hi!\nSyslog-ng received the following log message:\n $MSG\n-- \nsyslog-ng\n")

    );

};

This destination is named d_smtp. The smtp() function takes options for the mail server, port, from, to, message subject. and message body. Rather than worry about spam filters and mail rules, we’ll set up this destination to deliver mail on the local host.

Last, set up a new log rule.

log { source(s_src); filter(f_debug); destination(d_debug); };

log { source(s_src); filter(f_debug); destination(d_smtp); };

You added this rule to the existing one, so debug messages will go to file and to email. Execute the same test as before, and in addition to an entry in the file, you’ll see an email. You can read it with mail or mailx.

Return-Path: <noreply@example.com>

X-Original-To: egoebelbecker@localhost

Delivered-To: egoebelbecker@localhost

Received: from hala (localhost [127.0.0.1])

      by hala (Postfix) with ESMTP id 4062C602E32

      for <egoebelbecker@localhost>; Mon,  7 Sep 2020 12:05:08 -0400 (EDT)

X-Mailer: syslog-ng 3.13.2

Date: Mon, 7 Sep 2020 12:05:08 -0400

From: noreply@example.com

Message-Id: <1599494708.264239.11871@hala>

To: "Admin #1" <egoebelbecker@localhost>

Subject: [DEBUG] Debug log message of debug condition received from hala/egoebelbecker!




Hi!

Syslog-ng received the following log message:

 This is a test.

--

syslog-ng

Additional Destinations

Adding a new destination for a log message is very simple with syslog-ng configuration.

For example, adding Loggly as a destination is simply a matter of adding a LogglyFormat and Loggly destination.

template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [TOKEN@41058 tag=\"TAG\" ] $MSG\n"); 

  template_escape(no);

};




destination d_loggly {

  tcp("logs-01.loggly.com" port(514) template(LogglyFormat));

};

Now you can add a new log rule with this destination and the appropriate filters.

Add Syslog-ng to Your Logging Toolbox

You’ve learned how to configure syslog-ng and how to set up a basic logging system. You used the logger command line tool to test logging and then added email as an alternative place to send log messages.

Finally, you took a quick look at how Loggly can be integrated with this open-source logging tool. Keep working on setting up your system with syslog-ng and see what else you can make it do.

If you want to test SolarWinds Loggly with syslog-ng, sign up for a trial here.

This post was written by Eric Goebelbecker. Eric has worked in the financial markets in New York City for 25 years, developing infrastructure for market data and financial information exchange (FIX) protocol networks. He loves to talk about what makes teams effective (or not so effective!).