PagerDuty, Loggly, and Alert Birds
Well, I'm back, and this time I'm here to talk about an awesome product that we use all the time, PagerDuty. We use it internally for our own alerting (as do a number of Fortune 500 companies along with a million other startups), but we've also integrated it into Alert Birds, which is our alerting tool. With Alert Birds, you can configure saved searches that run against Loggly, and you'll run those searches over a period of time that you've selected, and Alert Birds will escalate alerts in PagerDuty. Before you can do any of those things, however, you need to set up the PagerDuty endpoint:
After you've done that, the next thing you'll need to do is to configure a saved search, and then configure the alert that you want to run. The search itself is pretty straightforward, it has a name, a search string e.g.
(this is why it's cool to send us JSON!), and a list of inputs and devices that you choose - you may want to run a particular search on only your web servers, for instance. The interesting bit is the alert itself, which runs a search that you choose, but has a number of options as to what conditions consitute an alert, and what the message should be:
This is where PagerDuty comes in. Although you can send a GET or POST request to an endpoint of your choosing with the alert data, triggering an alert in PagerDuty is far more useful, as they can SMS/email/phone you, and they also handle escalations and reporting. So, in the example above, if my web servers are spewing 500 exceptions, I want my ops folks to get notified, provided there are more than 10 - I don't want to wake anyone up over a little blip! I'm a nice IT manager like that. Anyhow, once an alert is in a critical state, it will run your search every minute until you're below the threshold, and once that happens, Alert Birds will automatically resolve your alert in PagerDuty.
That's pretty much all there is to it! You can find the docs on Alert Birds here, please do drop me a line at support@loggly.com if you need a hand, and until next time, happy alerting!