HAProxy Logging Using Syslog

Many things can happen when a user lands on a website. There are even times when user requests don’t reach a server, potentially giving you a false positive saying everything is OK. Long gone are the days when a website used only one server. In today’s distributed world, there are many servers behind a load balancer distributing the website user load.

So how can you know if everything’s running smoothly between your users and your servers? The logs emitted by load balancers and proxy servers like HAProxy give you a different perspective. They can tell you, for instance, how much time user requests take and how many requests fail. This information is valuable and can give you more clues when you’re troubleshooting incidents in a live environment.

In today’s post, I’ll give you a brief explanation of how logging works in HAProxy, how logging configuration works, how to process logs, and how to avoid running out of storage in your servers due to large log file sizes.

How Does Logging Work in HAProxy?

When you install HAProxy, you can configure how you want to work with the logs this tool records. For instance, you can configure HAProxy to send logs to different locations at the same time. Perhaps one team needs to get the logs in real time and another can use logs to identify a historical trend. Alternatively, you might simply want to migrate logs to a different location like a centralized logging storage solution.

Typically, when you configure logging for a tool, you want the logs recorded on a local disk. However, in a scenario where there are going to be many log-writing operations, this can be counterproductive. Performance might be degraded, and you might miss some of the logs. By default, HAProxy comes with a configuration to write logs to a socket because it’s faster than sending logs over the network or writing directly to a file.

In the HAProxy documentation, there’s a section covering logging in more detail. Once you have the basics from this post, you can quickly dive deeper into other topics like how to capture cookies or HTTP headers. You can even configure the type or verbosity of the logs you’d like to see.

If you haven’t done it yet, install HAProxy. In Ubuntu, you can do it this way:

$ sudo apt-get install haproxy

$ haproxy -v

HA-Proxy version 1.8.8-1ubuntu0.11 2020/06/22

Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Configuring Logging for Syslog

By default, there are two sections in the HAProxy configuration file: global and defaults. In the global section, you configure low-level features like security, performance, and logging destinations. The defaults section is where you have settings that apply to all other proxy sections below, like the front end and the back end. The goal is to avoid repetition. A post on the HAProxy website provides more detail about each section, but for our case, the global and defaults sections are enough.

In the default configuration file (/etc/haproxy/haproxy.cfg), you have something like this for logging:

global

      log /dev/log      local0

      log /dev/log      local1 notice

      # ... other properties not included here

defaults

      log   global

      # ... other properties not included here

When you look at the “/dev/log/” directory, you’ll see a symbolic link to systemd’s journal, which means logs are being sent to syslog:

$ ls -la /dev/log

lrwxrwxrwx 1 root root 28 Aug 21 22:37 /dev/log -> /run/systemd/journal/dev-log

Recent versions like Ubuntu 18.04 come with rsyslog installed. Otherwise, you’ll need to install it by running the command “sudo apt install -y rsyslog” in your server. Then look at the configuration at the rsyslog folder. In my case, it looks like this:

 ls -la /etc/rsyslog.d/

total 24

drwxr-xr-x  2 root root 4096 Aug 21 22:41 .

drwxr-xr-x 90 root root 4096 Aug 21 22:41 ..

-rw-r--r--  1 root root  314 Aug 15  2017 20-ufw.conf

-rw-r--r--  1 root root  255 Jun  3 02:08 21-cloudinit.conf

-rw-r--r--  1 root root  282 Jun 22 08:41 49-haproxy.conf

-rw-r--r--  1 root root 1124 Jan 30  2018 50-default.conf

Notice there’s a file with the name “49-haproxy.conf,” which is where the configuration for HAProxy logging exists. If you don’t see a file like this, create it. Here’s the default content:

$ cat /etc/rsyslog.d/49-haproxy.conf

# Create an additional socket in haproxy's chroot in order to allow logging via

# /dev/log to chroot'ed HAProxy processes

$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile

if $programname startswith 'haproxy' then /var/log/haproxy.log

&~

You can see logs are going to be written at the file /var/log/haproxy.log. From there, it’s your job to do something with those logs. However, you might need to restart rsyslog and HAProxy before you can see some logs. You can do this by running the following commands:

sudo service rsyslog restart

sudo service haproxy restart

You should at least see some initial logs like this:

$ cat /var/log/haproxy.log

Aug 21 22:42:44 ip-172-31-43-203 haproxy[1814]: [WARNING] 233/154135 (1814) : Exiting Master process...

Aug 21 22:42:44 ip-172-31-43-203 haproxy[1814]: [ALERT] 233/154135 (1814) : Current worker 1815 exited with code 143

Aug 21 22:42:44 ip-172-31-43-203 haproxy[1814]: [WARNING] 233/154135 (1814) : All workers exited. Exiting... (143)

Now that you have logs, it’s time to do something with them. It doesn’t matter which tool you use for processing your logs from the haproxy.log file. For instance, you can choose a tool like SolarWinds^® Loggly^®, which also has an agent to process logs and send them to Loggly automatically.

Configure Back-End Servers

If you want to give it a try to see how logging works by default when you add more proxies, you can do so by adding a configuration like the following at the end of the haproxy.conf file:

backend webservers

balance roundrobin

server webserver1 Your-Webserver1-IP:80 check

server webserver2 Your-Webserver2-IP:80 check

option httpchk

Replace “Your-Webserver1-IP” and “Your-Webserver2-IP” with the IP addresses of the servers you want to redirect traffic to. Save the changes and restart the HAProxy service. Notice we didn’t have to include any logging configuration in the back-end proxy section. You should now be able to send traffic to those back-end servers through HAProxy, and the logs will automatically go to the haproxy.log file.

Rotate Logs With Logrotate

Finally, as good citizens, we need to make sure the haproxy.log doesn’t eat all the server storage. To do so, you can use logrotate. In my case, it comes installed by default, but you can install it using the following command:

sudo apt-get install logrotate

Then you’ll have a configuration file like the following to clean up and compress the haproxy.log file:

cat /etc/logrotate.d/haproxy

/var/log/haproxy.log {

    daily

    rotate 52

    missingok

    notifempty

    compress

    delaycompress

    postrotate

        invoke-rc.d rsyslog rotate >/dev/null 2>&1 || true

    endscript

}

When you use the above configuration, logrotate will apply this rule every day, keeping any logs 52 days old or newer. It also compresses the rotated files into a gzip format. You won’t need to worry about having too many log files on the server—logrotate will remove any older files for you. By using this simple configuration, you can avoid having to wake up in the middle of the night to remove logs. Sounds like a great deal, right?

To make sure the configuration works, restart HAProxy and rsyslog:

sudo service rsyslog restart

sudo service haproxy restart

Conclusion

I only scratched the surface of the logging options HAProxy provides. You can continue looking at the official documentation. There’s also an excellent post on the HAProxy blog that goes deeper into how to change the format logs and even how to get profiling metrics.

Notice I used an Ubuntu 18.04 server in AWS, and most of the configurations I discussed here are there by default. Not every server will come this way, but at least now you know how logging works in HAProxy and where these configurations are so you can modify them if needed. Happy logging!

This post was written by Christian Meléndez. Christian is a technologist who started as a software developer and has more recently become a cloud architect focused on implementing continuous delivery pipelines with applications in several flavors, including .NET, Node.js, and Java, often using Docker containers.