Coolcam Your Logs with Geckoboard
Nothing is more addictive than good dose of data visualization. Yesterday I stumbled across Geckoboard and I have to confess I’m totally hooked. I’ve managed to bang out a simple AppEngine-based framework to proxy Geckoboard data from Loggly’s APIs. The code is currently feeding a logging dashboard on Geckoboard where we’re tracking 404 errors, signups, wiki edits, and searches from tweets about Apache logs.
To build your own logging mashup you’ll need an AppEngine account, a Geckoboard account, and a Loggly account. As of this writing, both Geckoboard and Loggly are in private beta, but if you tweet “Hey @loggly, I’m a natural @geckoboard hacker!”, I’ll make certain you get a Loggly invite. Maybe the guys over at Geckoboard will do the same!
Once you get your accounts all sorted around, start by creating an HTTP input called ‘appengine’ in your Loggly account. You’ll use this input to log from your Python code on AppEngine to Loggly. Follow the directions on the wiki to create an HTTP input if you need help.
If you run your own webserver, you’ll also need to configure your syslog daemon to send us your web application logs. Again, follow the instructions on the wiki to setup file monitoring. We recommend using the latest version of Syslog-NG to monitor and forward log files to Loggly.
Keep in mind you could do all this from just AppEngine if you are already hosting your app there. Check out the logging from Google AppEngine section on the wiki for more info on logging from AppEngine into Loggly. The current code is asynchronous and non-blocking!
Getting the AppEngine code running is pretty straightforward. Go grab the code from Github, and then save it into your local AppEngine Launcher. Edit the app.yaml file and change the name of the app to something unique. Log into your Google AppEgngine account and create a new application of the same name.
Now go edit the main.py file, and edit up a few lines:
- the appengineurl should be your appname + .appspot.com
- the logglyaccount should be your loggly account name + .loggly.com
- the instances of username/password should be the same as your Loggly user’s credentials
- the URL in the logging setup (at the bottom) should be a URL for one of your HTTP inputs
Once you are done editing, start up the AppEngine Launcher and go to File…Add Existing Application and add the new application to the launcher. Feel free to test the app locally first to ensure it’s working correctly. You’ll get a few links to test with, which will return XML.
Once you get all this working, head on over to Geckoboard, and create a new custom line chart widget. Set the widget type to custom and the format to XML. Enter a label for the chart, and then enter a URL with which Geckoboard will fetch the data from Loggly.
Edit the URL a bit to reflect the app name you picked on AppEngine, and the term you are using to search your logs. In our example, our app name is ‘logglygecko’ and we’re looking for ‘404’ across all inputs for the last hour and day.
If you want to include additional search terms, you’ll need to hand encode the URL’s spaces. This example looks for 404 errors on .jpg files:
The code also supports the meter widget. Just create a meter widget on your dashboard, and go through the same settings as for the chart widgets, except replace graph with meter in the URL.
The Sky’s the Limit
Geckoboard already has a ton of widgets for displaying valuable data from various services like Google Analytics, MailChimp and more. With Geckoboard’s custom widgets, and the data coming in from Loggly’s APIs, you can now quickly hack up a dashboard to display all that gold hidden away in your log files too!