Loggly

Loggly's Outage for December 19th

Posted 19 Dec, 2011 by Kord Campbell in Business and Startup

Sometimes there's just no other way to say  "we're down" than just admitting you screwed up and are down.  We're coming back up now, and in theory by the time this is read, we'll be serving the app again normally.  There will be a good amount of time until we can rebuild the indexes for historic data of our paid customers. This is our largest outage to date, and I'm not at all proud of it.

So What Happened?

Sometime yesterday afternoon ALL of our machines on Amazon's East region, availability zone 1d, were rebooted by AWS staff.  Originally we stated we had not received reboot notices from Amazon, but the truth is that (4) of the staff here, myself included received two separate vague notices, one from about 10 days ago, and another from 3 days ago, which stated 'some or all' of our instances were scheduled to be rebooted.  These notices were found in our spam folders on Gmail, placed there with a very large red notice reading: "Warning: This message may not be from whom it claims to be. Beware of following any links in it or of providing the sender with any personal information."  Meh.

Loggly uses a variety of monitoring mechanisms to ensure our services are healthy.  These include, but are not limited to, extensive monitoring with Nagios, external monitors like Zerigo, and using a slew of our own API calls for monitoring for errors in our logs.  When the mass reboot occurred we failed to alert because a) our monitoring server was rebooted and failed to complete the boot cycle, b) the external monitors were only set to test for pings and established connections to syslog and http (more about that in a moment), and c) the custom API calls using us were no longer running because we were down.

Combined, these failures effectively  prevented us from noticing we were down.  This in of itself is was the cause of at least half our down time, and to me, the most unacceptable part of this whole situation.

The Human Element

The other cause to our failures is what some of you on Twitter are calling "a failure to architect for the cloud".  I would refine that a bit to say "a failure to architect for a bunch of guys randomly rebooting 100% of your boxes".  A reboot of all boxes has never been tested at Loggly before.  It's a test we've failed completely as of today.  We've been told by Amazon they actually had to work hard at rebooting a few of our instances, and one scrappy little box actually survived their reboot wrath.

While some might go on a rant about how 'normal' failures don't affect 100% of your boxes the truth is that any and everything (including an army of reboot monkeys) can be expected to happen to your servers if you wait around long enough.  The trick to being good at running a reliable service is to architect around any number of everythings that could happen to your service and build for it.

In this case we didn't ever build the workaround simply because the system we run - a combination of 0MQ+Solr+Zookeeper+Loggly Special Sauce - makes it extremely challenging to survive a complete failure with more than 1/2 of the cluster missing.  With other challenges facing us, we decided to live with the risk. Now we're dealing with the fallout of our decision.

So, How Do We Make This Right?

Single instances of Loggly's search cluster can't be spread across multiple availability zones or regions due to the amount of data we push around, latencies between the search nodes, and the lack of support in our system for redundant indexes.  We've been OK with those limitations in the past simple because we normally archive data to S3 when we catch it, and we are capable of rebuilding indexes on the fly if we lose one or more indexers.

The first step in addressing this is to start sharding our customers across multiple Loggly deployments.  This will prevent further outages to the entire customer base.  The second step is to start doing Loggly deployments on dedicated hardware.  Because we keep large amounts of data on our boxes (the indexes) this is pretty much a requirement for fast recovery times when a deployment goes down.  While S3 is AWESOME for backups, it sucks big-time for rebuilding a large amount of search index data.

The second step is to ensure more robust external monitoring.  With multiple deployments, this issue becomes less of an issue, but clearly we need more reliable checks than what we rely on with Zerigo or other services.  Sorry, but simple HTTP checks, pings and established connections to a box do not guarantee it's up!

Finally, we accept full responsibility for the impact to our customers.  We will be in touch with our paid customers sometime over the next week to address compensation for this outage.

We welcome feedback below, and encourage useful criticism of our architectural choices.  All I would ask is that you consider Loggly's infrastructure isn't the same as yours, and I've greatly simplified the reasons for not being more redundant in our deployments.  We can, and will, endeavor to do better in the future.

Kord Campbell, CEO

Tweet