Automagically logging from EC2 using Loggly– 1 server or 1,000

As almost everyone knows by now the “E” in AWS EC2 stands for “elastic”. EC2 Instances come and go, and if one uses EC2 Autoscaling, Instances really do come online without any manual intervention. This allows us, as engineers, to build systems that can grow automatically as our needs grow.

In this blog post I’ll show you how you can have your EC2 Instance automatically send its logs to Loggly using rsyslog, without any intervention on your part. The moment the Instance comes online, its logs will be appearing in your Loggly account — and, importantly, you’ll have those logs even after the Instance terminates. You can even set up an Alert in Loggly, letting you know a new Instance has come up!

Don’t put your credentials in your AMI

AWS considers it bad practice to place any sensitive information in an AMI you create. What if the AMI is accessed by an authorized person? Then those credentials could be compromised. In the case of Loggly this means your Customer Token. Fortunately AWS provides a mechanism to allow you to securely, and automatically, supply your Customer Token to an Instance on boot-up — use Instance user-data.

Configuring rsyslog on Instance boot-up

To have your Instances automatically configure rsyslog, install this script at /etc/init.d/config-loggly. Then symlink it from /etc/rc2.d/S22loggly:

root@ip-10-147-169-175:~# ls -al /etc/rc2.d/S22loggly
lrwxrwxrwx 1 root root 23 Sep 18 14:40 /etc/rc2.d/S22loggly -> ../init.d/config-loggly

With this in place the script runs automatically at boot-up. How does it work? The script starts by reading in your Loggly Customer Token, which is passed in via user-data (see below). It then takes this Token, and creates a suitable rsyslog template so that all logs from this Instance are sent to Loggly. Finally it restarts rsyslog (newer versions of rsyslog require this) so this template takes effect. That’s it! This scales wonderfully, whether 1 Instance is launched from the AMI, or 1,000.

To ensure all future EC2 Instances have this in place, create a new AMI from this EC2 Instance, and launch all Instance using this new AMI. Instructions on how to create your own AMIs can be found on the AWS website.

Setting the user-data

Of course, your Customer Token must actually be passed into the Instance, so it can be retrieved by the script. This can be done manually when launching an Instance. For example, the AWS Management Console allows you to set user-data when launching the Instance. Shown below is how I set the Customer Token to 727bae3d-750c-4e44-96ed-ad83230208bb.

Setting user-data is supported by many tools, including frameworks such as boto, allowing you to script this. Autoscaling also supports passing user-data to any Instance launched by that service.

Other Approaches

Of course, there are other approaches. Your favourite automation tool could also drop the rsyslog template file into /etc/rsyslog.d/ and restart rsyslog after the Instance starts. This avoids needing to create a custom AMI. So set up automatic logging, and don’t ever lose those logs again.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.