Loggly

At Loggly, we're not your father's IT company

Posted 1 Apr, 2013 by dave@loggly.com in Business, Log Management, and Startup

A few days ago an editor from CIO.com asked Loggly to provide two tips on hiring/retaining engineering talent for a future article.  “Sure -- happy to help”, I replied.  I then mentally outlined how we build and keep our highly sought after development team happy (out of 24 employees 25 employees, another just signed, 22 of our hires write code).  The standard Silicon Valley startup checklist popped up:

• Competitive salary, equity packages, etc
• Open vacation, flex hours, etc
• Catered lunches, snack bars, and top-shelf beverages
• Latest-greatest technology and development environments

• Work on a product that thrives on big data for a formidable challenge ($100b market, per IDC)
• Solve a true market created big problem, that matters... and one that people pay for
• Make a difference, and be recognized for it.

"Thanks for the tips... Unfortunately, they weren't quite right for the article.
(Every IT guy I have ever known has DREADED being asked to man the booth or attend trade shows and conferences)."

• Building an old-school IT product (legacy deployment cycles aimed at non-cloud savvy companies run by stiff CIOs)
• Selling an old-school IT product (hundreds of end-users who are also our buyers sign up WEEKLY)
• Selling into the CIO and expecting our technology to be forced down (vs adopted, loved and shared)
• Looking, hiring or retaining old school ‘Dilbertized IT guy' engineering talent who hides at the mere thought of customer interaction

• The billion dollar opportunity we are solving in log management (the booth was busy!)
• The ability to hear from customers and prospects exactly what they want in a product and how they are using it (the good and the bad)
• The way they can make a difference with their efforts to improve and continually optimize the product (our customers enjoyed the face-to-face conversations)

• Software Developer
• Sales Engineer
• Director of Product Marketing

No Comments   |   Leave a Comment   |   Tweet

Building Great Software...5 Cool Tips For Success!

Posted 15 Mar, 2012 by Brian Schroeder in Business and Startup

Writing software is hard. Writing it well is even harder. I have been rolling some ideas around in my head on how to make software that is better for the people creating it and the people using it. Software has some distinct (and not-so-distinct) phases that it goes through as you work on it. The first of those phases for developers is the design phase. Here are some ideas on making better software during the design process.

If you start with "you know what would be cool?", kill the idea.

OK, so maybe that's a little harsh, but that's my knee jerk reaction when I hear a "you know what would be cool" idea. Let me rephrase my suggestion; stop and think. You may be wondering what I have against doing cool things. Nothing. There's nothing wrong with doing cool things. The problem is that we developers gravitate toward using a cool piece of technology for the sake of using it rather than finding the best fit for our problem. We often attempt to solve a problem either using a cool new piece of technology in a way that it wasn't intended for, or new language features when an alternative would have been better suited. Redis, mongodb, rabbitmq, Python, Ruby, Clojure, Scala, Node.js are all great technologies that meet various needs in their own ways. Sometimes they may be the best fit for your problem, but not always. Just think twice when you think you're about to do something "cool". Developing software is less about being cool and more about building something that works well.

Aim for simplicity.

Software developers have a knack for building and stacking and inheriting and composing until the original simple solution is unrecognizable in a mound of clever engineering. When you're thinking about the next piece of code that you're going to write, start simple. Don't build something in your mind that is more complex than it has to be. You should always use the simplest thing that works, except for when you shouldn't. Next point.

The simplest solution isn't always best.

Sometimes the simplest thing just won't cut it. Sometimes you need that really weird class hierarchy or a really tricky algorithm or something that really hurts your brain to even think about. Usually this is going to happen on the second iteration of a problem. You should still always start with the simplest thing. If that doesn't seem to hold up, go on to something more complicated, but keep the complexity to a minimum.

Don't build more than you need.

It's often tempting to try to build a solution that handles every contingency or meets the needs a user doesn't even know what they have yet. When you are tempted to design everything into your application, think twice. Don't design what you don't need. If you can come up with a good reason for designing it in, then go for it. Otherwise, don't bother. Build just what you need and no more.

Think long-term.

I've seen so many developers (including myself) focus on getting something working now and neglect foreseeable issues with maintenance and deployment (which are the two major areas that this impacts). When you are thinking about the next piece of code that you are going to write, take the time to make sure you are focused on the long term solution. This may sound like it contradicts my previous idea of doing the simplest thing possible, but I think these ideas can co-exist. Start simple. Get something working. But don't plan on getting it "justworking"; plan on getting it working so that the next generation of developers that work on it will be happy you made the decisions you did rather than having them curse your name.

These aren't new thoughts. I've heard them stated by other experienced developers in various ways. And they're not immutable laws of nature. I think they're good guidelines, but the world won't come crashing down on your ears if you do something different. But I do believe that your experience of designing software will benefit from it. And as a result, your software will be better for you and your customer.

2 Comments   |   Leave a Comment   |   Tweet

Introducing the Loggly Wood Shop

Posted 2 Mar, 2012 by inga weizman in Startup

We've been talking about doing a Loggly swag store for a while, and I'm excited to announce it's finally here!  Now everybody everywhere has access to our snazzy swag.  We have some rocking t-shirts, stickers and beer glasses available for purchase.  I'm not just saying this because I work here, when I wear Loggly t-shirts out on weekends people compliment me on them.

We’ve always made a point to produce swag that has lasting appeal. All of us have gone to conferences and collected a bunch of stuff, only to take it back to our room, look at it, and leave it there. We are proud to say that people actually wear our swag out and about, track us down at events specifically to get their hands on it, and tweet at us to ask about how they can get it. We've even been offered money for the shirts off our backs, no joke.  We are incredibly lucky that the allure of our brand continues beyond our product. Even folks who’ve never used our product, and never will, love the branding and culture we’ve created and want to be a part of it. The Loggly store was born from this enthusiasm and support, and we’re extremely grateful for it. 

We’d like to give a special shout out to our fantastic illustrator Andre Jolicoeur who has done an outstanding job bringing Hoover the beaver and the Loggly brand to life with his creations. We also owe a great deal of praise to Shopify, whose service made it a breeze for us to get our store up and running. The Shopify Textmate bundle made development simple and faster than I had anticipated, and the entire launch process was demonstrated clearly through the Shopify interface. 

To celebrate our launch we’re offering 20% off all merchandise this week, redeemable by using the discount code beaverlicious at checkout. Happy shopping!

No Comments   |   Leave a Comment   |   Tweet

Partnering with Scalr Gets You a Margarita

Posted 23 Feb, 2012 by inga weizman in Business and Startup

We’ve all dreamed of doing our job while sipping a margarita on the beach, Loggly and Scalr can get you there...well almost.  We love things that scale and automate and all things cloud and that is why we partnered with Scalr.  Scalr is an open source cloud management tool that brings automation to web applications.  The cloud is all about scalability, growth, and making things easier to manage.  Scarl gives you the ability to manage as many servers as you need on different cloud computing services and adjusts load capacity as you have spikes and valleys in your traffic.  It’s like a system’s admin machine that never sleeps, doesn’t need energy drinks and coffee, so now your system’s admin can actually can have a life.  Loggly comes in and takes care of all the logs, we store them, make them searchable, as well as providing features such as monitoring,  troubleshooting and user analytics so nothing is lost and you can figure out what your users are doing.  Sit back, relax and see what Scalr can do for you. 

• DNS Management: automatically created and updated for you
• Fault tolerance: servers crash, at the worst times, now the problem is detected and automatically resolved for you 
• Multi-cloud deployments: no need to commit to just one, infrastructure can be spread out across multiple providers  
• Integrated SSH and key management

We're super excited to be partnered with such a cool company that's helping web apps scale and making your life easier. 

No Comments   |   Leave a Comment   |   Tweet

Big Data Uncovered?

Posted 7 Feb, 2012 by Kord Campbell in Business and Startup

I recently came across an eWeek article titled 2012: A Cloudy Year for Big Data by Frank Ohlhorst. You could easily say I have a few opinions on the matter of big data!  :)

First, I agree with Frank’s first notion that big data is neither big or new.  The fact is, I've been saying things like "Dude, that's a ton of data!" since I started notching out the opposite sides of floppies back in the 80s.  Remember these?

Ohlhprst quickly follows up his vague handwaving that ‘big data’ a new term with, “For most of its existence, big data has been out of the reach of small and midsize businesses (SMBs) because the storage and processing power needed to make this technology work is too expensive.” 

Companies have been doing for years what they need to do better business, regardless of whether or not it’s expensive. In manufacturing, the costs of a small company optimizing on how to efficiently making tons of a cheap product can actually be quite a bit more expensive than a larger company making a few units of a complex product.  In the same vein, smaller business may have more complex business optimization processes than larger ones, and require relatively larger amounts of data are required to solve those problems than with larger companies.

I agree with Frank that small business typically don't always have the resources necessary to solve massive scale problems, but again the problems are relative.  For example, small software startups don't have project managers where larger ones do, not because they can’t afford them, but because they really don’t need them in a full-time capacity.    I think this may be part of why SaaS services have been a huge hit and the term cloud has taken off because of it.  SaaS allows companies to tackle a wide variety of problems across the entire business, all the while providing cost effective high tech solutions to solve problems in a way you could never have done before.  For the first time in history the quality of business processes is experiencing sustained growth.

“These new cloud-based capabilities are on a growth path and are creating more opportunities for even the smallest of businesses to leverage big data without the traditional expenses of compute farms and massive storage arrays.” 

Yes.  However, compute farms and storage aren't the main thing that these companies need. They need access to the raw data that contains the data about their business, and the tools to extract the data in which they can take action. Figuring out your company’s problems requires brain power, understanding, data and tools. CPU and disks don't solve complex problems. People do.

It's All About Application Analytics

Ohlhprst also describes big data analytics as being comprised of  three primary elements: volumes of unstructured data, processing power and algorithms. However, big data doesn't always imply unstructured data. Log files, or what Loggly calls Big Time Data™ typcially contain a large amount of structure.  Dealing with structured data isn't always easy, and if you write software that 'expects' a certain structured format, your analysis can sometimes be broken or flawed if it encounters data that doesn't fit the structure you coded for.  One way around this problem is to apply extra meta data to the data set.  One technique to solving to this problem is adding a search index to the data, which is the approach Google pioniered and what Splunk and Loggly do for log files or time series event data.  By being able to do text search data, and interact with it in realtime, or near real time, the user can optimize on solving the problem 

Ohlhorst continues, “For it to be true big data, there has to be lots of it, and most SMBs don’t generate that volume of data internally, which leads them to seek out alternative data sources. Here, the cloud delivers.” Not true.  Big data should be defined as an amount of data that a human can not reasonably digest. Generating large amounts of meaningful data is actually a bigger problem. Again, it's understanding the problem you have before you can solve it.

Yup. Ohlhorst explains that throughout 2012, data sets and others can be expected to grow exponentially - “The amount of data being generated globally increases by 40 percent a year", according to the McKinsey Global Institute, a data analytics research firm.  True.  The access of this data, mostly through the web, generates vast amounts of data as well. Ohlhorst continues that information needs to be organized, sorted and processed- and that takes computing power.  Frankly nowadays, CPU is cheap enough that most of these problems can be solved on your laptop. Fast CPUs for crunching 'big data' aren't the problem any more than a search engine's main problem is crawling for data. The real bottleneck is adding meaning to the data that a customer can digest and make actionable.

PaaS/IaaS Accessibility Is a Problem 

I’m glad that Ohlhprst recognized that Amazon isn’t the only one in the game in offering private cloud-based big data analytics platforms. He believes that since this technology is designed as a complete platform and not as a service, these platforms are still out of the reach of the SMB market.

Ohlhorst is right that these platforms are out of reach - but not just because they are designed as a complete platform and not as a service. I think it's because SMBs don't know they need it, don't have the data to put on it, and don't have the resources to manage it.  There are plenty of hosted solutions out there (see SalesForce and their app marketplace) that provide some serious horsepower to the most important task - managing a company's contacts.

And of course, there had to be a Splunk mention in his article.  Splunk sells expensive enterprise software.  Their software is often times the most expensive piece of software a company has ever bought. Sounds like Oracle, eh?  They aren't converting big data analytics into cloud services; they are simply taking their product and making a slimmed down version into a cloud offering they can generate leads with.  Any serious big data customer they land will have to buy that very expensive solution and install it on a bank of computers and then pay people to manage it.  SaaS is not what Splunk is taking to the market when they go IPO.  It's their hellaciously expensive software licenses.

Big Time Data™.  It's in the future of your small business.

No Comments   |   Leave a Comment   |   Tweet

Getting Your Product Sticky

Posted 13 Jan, 2012 by Kord Campbell in Log Management and Startup

A few months ago I made an off-the-cuff remark about Loggly. "We're like one of those shitty solar powered calculators.  When it gets dark, we forget everything you've typed into us."

That comment wasn't far off the mark.  Historically, we haven't provided a whole hell of a lot of features that makes it easy to jump back into where you left off on your last search session.  Basically when you logged out of Loggly, or even closed the shell in your browser, we'd forget everything you searched for until that point.  It made it extremely difficult to get back to something meaningful the next time you logged in.

We shouldn't be here if we aren't meaningful.  We should deliver users a 'punch in the gut' feature that makes a lasting impact.  One they don't want to avoid.

Saving Time with Sticky Features

Scaling search for a massive amount of log file data being sent in from thousands of machines has been an overwhelming non-trival problem to solve for us over the past year, and it's been our top priority.  Unfortunately us solving scale issues aren't readily obvious to users.  Users always expect things on the web to be fast.  They could care less how hard a problem it was to solve.  They don't think to themselves, "Wow, that's fucking fast!".  No.  Instead they sit around and mutter things to themselves like "Why the hell doesn't feature X do Y?  This thing is wasting my time!".

And there it is laid bare: Don't waste your user's time.  It's the most valuable resource they have.  Get to the point quick, make it easy to get back to what you were doing next time, and do it all with little fuss and muss.

I say give them sticky features!

Saved Search and More

And so, without further ado, I'm officially announcing one of many-to-come new sticky features: saved search.  Saved search provides users a way to write a search query and then preserve the search to run again later.  Saved searches can generate facet graphs or they can simply run a regular search across a given time range.

The shell has been reworked to provide context changes to use with the saved search feature.  You can now change the date context, or limit the context to certain inputs, and then rerun the search or graph using the red rerun button at the top.

Here's a quick screencast running through some of our new sticky features:

Coming Up Soon

We're continuing to add features that increase stickiness to the product.  Next week we'll be releasing a revamped history feature for the shell page, where what you've typed in before in a session will be preserved in your command history, just like it would in a normal shell prompt.  We're also adding customized graph selection on the main dashboard, which will allow you to start viewing events that matter most to you by default when you first log in.

All these featuers are leading up to a major revamp of the way we provide value for our user's events.  Expect completely customized dashboards for server monitoring, website performance, user analytics, and more soon!  If you have a feature you'd like to see us implement, please do drop us a line.  We're keen on not wasting your time!

No Comments   |   Leave a Comment   |   Tweet

Loggly's Outage for December 19th

Posted 19 Dec, 2011 by Kord Campbell in Business and Startup

Sometimes there's just no other way to say  "we're down" than just admitting you screwed up and are down.  We're in the process of rebuilding the indexes of historic data of our paid customers. This is our largest outage to date, and I'm not at all proud of it.

So What Happened?

Yesterday afternoon all of our machines on Amazon's East region, availability zone 1d, were rebooted by AWS staff for maintenance purposes. 

The cause of our failure is what some of you on Twitter are calling "a failure to architect for the cloud".  I would refine that a bit to say "a failure to architect for a bunch of guys randomly rebooting 100% of your boxes".  We've been told by Amazon they actually had to work hard at rebooting a few of our instances, and one scrappy little box actually survived their reboot wrath.

While some might go on a rant about how 'normal' failures don't affect 100% of your boxes the truth is that any and everything (including an army of reboot monkeys) can be expected to happen to your servers if you wait around long enough.  The trick to being good at running a reliable service is to architect around any number of everythings that could happen to your service and build for it.

In this case we didn't build the workaround simply because the system we run - a combination of 0MQ+Solr+Zookeeper+Loggly Special Sauce - makes it extremely challenging to survive a complete failure with more than 1/2 of the cluster missing.  With other challenges facing us, we decided to live with the risk.

So, How Do We Make This Right?

Single instances of Loggly's search cluster can't be spread across multiple availability zones or regions due to the amount of data we push around, latencies between the search nodes, and the lack of support in our system for redundant indexes.  We've been OK with those limitations in the past simple because we systematically archive data to S3 when it arrives and we are capable of rebuilding indexes on the fly if we lose one or more indexers.

Our primary method to address this will be to start sharding our customers across multiple Loggly deployments.  This will prevent further outages to the entire customer base.  We've already been investigating other data centers on both dedicated hardware and other cloud-based services.

Finally, we accept full responsibility for the impact to our customers.  We will be in touch with our paid customers sometime over the next week to address compensation for this outage.

We welcome feedback below.

Kord Campbell, CEO

3 Comments   |   Leave a Comment   |   Tweet

Java Code Coverage With Cobertura and Jenkins

Posted 29 Nov, 2011 by Noah Gift in Code and Startup

Introduction

Cobertura

Jenkins

Conclusion

No Comments   |   Leave a Comment   |   Tweet

Automated Integration

Posted 16 Nov, 2011 by Mike Blume in Business, Code, and Startup

One of the fundamental challenges of distributed coding is deciding what/when to integrate. Sure, that patch your colleague just sent you looks good, but is it actually ready to go into master? At Loggly, we've been feeling our way towards a disciplined integration process. A year ago, our frontend developers were all making commits directly to trunk in a single SVN repo. Once every few weeks, we'd run `svn up` on our servers, and hope for the best. Today our code goes through peer review, unit testing, and static analysis before it even touches our master branch.

Like most projects these days, the process starts on github. Fork. Push a feature branch to your repo. Open a pull request. Go through a couple rounds of discussion and revision. Merge. Every change to our code goes through this process. At first we thought it would slow us down, that we'd want pull requests for the nontrivial code and to just push to master for the easy stuff. After just a few days, we found the pull requests were slowing us down not at all, and that we all enjoyed the greater transparency into our colleagues' work.

Once we merge, the automation kicks in -- our default integration branch is 'proposed', so clicking merge doesn't actually get the code into the master branch. Jenkins polls our 'proposed' branch once a minute, then runs a simple preflight script on the code.

Rather than keep that preflight in a jenkins configuration page, we have it checked into the codebase so that any developer can run it too; this way there's no excuse for breaking the build -- you should have seen it break locally =P

Here's our preflight script. Let's go through it line by line.

DIR="$( cd "$( dirname "$0" )" && pwd )"

APP=$DIR/..

First we figure out where we're running, so that we can find the other scripts distributed with the app. 

$DIR/purge_pyc

Next we purge pyc files. This is done because if a user recently switched from a branch which contained files which don't exist in our branch, the pyc files may still be around, and may be found by the interpreter. 

$DIR/syncenv

Next we run a script to sync our python virtual environment, and ensure all requirements are present. 

$DIR/runtests

Here, of course, we run our unit tests. Each run prints a coverage report, so that as we recover from our testing debt, we can measure our progress.

&& $DIR/lint...

Next, and this is important, we run pylint over the parts of our app that we expect to pass with no warnings. As we clean up our app, we continue to add modules to this list. Pylint does a few useful things for us. It looks for trivial name errors of the kind that could quickly cause code to stacktrace -- using a module without importing it, etc. It also enforces certain kinds of coding discipline. Our functions and modules can't exceed a certain length. The cyclomatic complexity of our functions is limited. 

If all of this passes successfully, Jenkins automatically pushes the checked-out commit to master, which is where we base our development. Thus, we're always basing our development on known-vetted code.

If any of it fails, Jenkins still has a couple more tricks to pull. Here's our on-failure script:

This comes in two parts. The first runs a standard-issue git-bisect between origin/proposed and origin/master. Since origin/master has already been vetted by jenkins (that's how it became master), we know there'll be a regression somewhere between the commits. This goes into the session output, and is e-mailed to the relevant committers. Next, we roll the proposed branch back to the already-vetted master branch. Whatever pull request broke the build will have to be re-made from scratch.

No Comments   |   Leave a Comment   |   Tweet

Get Real with Pusher! Getting Those Alert Birds Squawking...

Posted 4 Oct, 2011 by Brian Schroeder in Code and Startup

•     Listening for events.
•     Triggering events.

No Comments   |   Leave a Comment   |   Tweet

Alerting with Alert Birds

Posted 13 Sep, 2011 by Kord Campbell in Log Management and Startup

Back during my ISP heydays in the late 90s, I stumbled across a project called Peep The Network Auralizer.  Peep basically takes things like netflow events from remote clients and sends them to a audio enabled server.  Network traffic sounds like running water.  Pings sound like birds chirping. SSH logins like frog croaks.  The result is something that sounds like one of those sleep machines from the 80s.

What was really interesting about Peep, besides the fact it could lull you to sleep, was it could notify you of events that were important in aggregate but not necessaraly by themselves.  A ton of failed SSH logins became a chorus of servers crying for help.  High network load became a rushing river.

Now I look back on it, my use of Peep was my first pass at alerting and monitoring complex events in my infrastructure.  Today it's also a good excuse to build an app for Loggly which makes a bunch of stupid bird noises.

Alerting Rules

Alerting is the #1 Loggly customer requested feature, so much so in fact that I broke down a few months ago and wrote a post on doing alerting with Loggly and Pagerduty.

After I wrote the post one of our customers commented that it sucked he had to run his own server to execute the cron jobs which ran the searches on Loggly.  He's right.  Requiring your users to run yet more infrastructure just to use a core bit of your service isn't the best way to win favors.  And, it certainly doesn't scale well.

On that note, I'm pleased as punch to announce Loggly is now shipping alerting via the Alert Birds service.  You can follow the Alert Birds on Twitter too!

Using Alert Birds with Loggly

Using Alert Birds with your Loggly account is simple.  Navigate to the Alert Birds website and then log into your Google Account.  Once you are logged in, you'll be given a form that allows you to enter your Loggly's account 'subdomain'.  Once you submit your subdomain, you'll be taken to Loggly to login to that account and grant access to the Alert Birds' app.  You'll be taken back to the Alert Birds' site when you are done authing the app to Loggly.

Alertbirds allows you to create an unlimited number of alerts, each of which are comprised of saved searches run on Loggly and endpoints that get called when searches return above or below a given threshold.  Alerts are run on a schedule you define, and the endpoints currently (only) support pushing events to PagerDuty's APIs for notification.  Alert Birds also allows you to trigger sound events in a  browser viewing the Alert Birds' website. 

Here's a short video on using the Alert Birds' application (sorry for saying um so much):

For more detailed setup info, you can reference our wiki page on Alert Birds.  If you'd like to trigger an alert for Kord on his blog, click here.

Apps for a Logging Platform

We've Open Sourced the Alert Birds project to help illustrate how easy it is to build a logging application on Loggly.  Alert Birds was writtien to run on Google's AppEngine service, but we'll be launching other apps soon that run on other frameworks and services.

Keep an eye out for more applications and app support coming out of Loggly in the near future.  We're working on apps like  a 3D globe for visualizing website traffic, web analytics, application reporting, and more!

No Comments   |   Leave a Comment   |   Tweet

Intern Alley Comes To a Close...

Posted 15 Aug, 2011 by inga weizman in Startup

We've had the pleasure of having 3 great interns, that were often referred to as Intern 1, 2 and 3 "working" here during the summer and here is what they had to say in their own words about the Loggly experience.  We miss you already!   

Josept Friedrich (Intern #1)

I've definitely learned a lot since telling Kord I wanted to be a VC. In an effort to save my soul from the business world I've enjoyed improving my VIM skills and dabbling in emacs.  School is an experience in and of itself but my time here at Loggly has given it a run for its money.  Things I will miss:

Richard Berwich (Intern #2)

What I did at Loggly?

Keving Wang (Intern #3)

1 Comment   |   Leave a Comment   |   Tweet

Bebo's Logging Story

Posted 9 May, 2011 by Kord Campbell in Business, Log Management, and Startup

Aren Sandersen, VP Operations for Bebo, came over today and had lunch with us. Afterwards, we sat down and chatted about how Bebo is changing their infrastructure, manages logs, and how they use Loggly to do debugging, alerting and operational troubleshooting with Loggly. You can view the video on your iPhone via broadband, or watch the mobile version as well.

You can follow Aren and Bebo on Twitter, or sign up for Bebo on their website.

1 Comment   |   Leave a Comment   |   Tweet

Everyone's Talking About AWS Being Down

Posted 22 Apr, 2011 by Kord Campbell in Business and Startup

Everyone seems to be blogging about how their service has ben impacted by Amazon’s AWS outage, or whining about how Amazon sucks, or explaining why their service was architected so well that it didn’t impact them, or why you suck if you didn’t plan for this.

As Loggly is based entirely on AWS and was only minimally impacted during the first few hours of the start of the outage, I figured I’d share exactly how we managed to do what we did:

We run across multiple availability zones and don’t rely on EBS for anything other than backups of a few simple databases. Everything else is file based the EC2 instances and their drives are set up in a RAID-1 configuration for speed and slightly more reliablity. Our log streams are backed up to S3 every few minutes as they come into our proxies. We rely on RDS for the database for the user logins, which really ended up being the only thing affected.

I asked Jordan Sissel, Head of Ops and Senior Developer here at Loggly to describe exactly what happened when Nagios/Pagerduty went off a night before last. Here’s what he said:

I saw RDS (prod db) problems in the early morning just as the problems started, but by the time I started debugging it the problem went away. I was notified by pagerduty because beaveroil and some other checks were failing.

Otherwise we weren’t really impacted. We got lucky, I think. I kept my eye on service but it stayed happy during the AWS outages.

Worst case, it’s easy for us (assuming rightscale is functioning, which it wasn’t for some of the day) to migrate to different parts of EC2 due to our use of puppet and are lack of EBS usage (only our RDS uses EBS)

Planning for Failure

Jordan is right, we can pretty much do a Loggly deployment on any AWS region within 20-30 minutes. Because we use Zerigo for DNS, and because we keep short TTLs, we can switch out records and have them updated quick and redirect ALL our inbound and outbound traffic to the new deployment.

Of course that leaves the question about migrating data on our existing or failed indexers. Thankfully that’s not a huge issue for us because we can rebuild them using EMR from our S3 backups.

Before we launched Loggly’s public service, I mandated that Loggly must be able to rebuild failed indexers at will. The work required to support this ended up delaying our public launch by at least 45 days. Now if we lose a box or have to move to another region, we can rebuild any (or all) of our indexers in at most a few hours. In theory we could continue to index new data coming into the system and historical search impact to customers would be minimal.

As my friend Clay Loveless put it so elegantly, “We may rip the rug out from under your feet at any moment.” If you haven’t planned for disaster striking, then you should go back and reassess your infrastructure. Hopefully we’ve planned well for just such a disaster.

<knocks on wood>

No Comments   |   Leave a Comment   |   Tweet

Server Density

Posted 23 Nov, 2010 by Kord Campbell in Business, Log Management, and Startup

My old friend David Myton from Boxed Ice swung by the Loggly office the other day to say howdy. I sat down with him and did a quick Geek CEO video about bootstrapping, developing product, filling the sales pipe, and listening to him being wise-beyond-his-years about raising capital.

Server Density is now growing at 20% a month, enjoyes a super low churn, and just got to break-even over the weekend. While others are just dreaming about startups, or grinding away for the man, David is here living the dream.

David Myton, CEO of Boxed Ice from Hoover on Vimeo.

David shared with me that the company will be moving to the Bay Area in a few more months, once they grow a little more and get their work stuff sorted around. They’re more than welcome to squat with us when they do!

No Comments   |   Leave a Comment   |   Tweet

Loggly Extended Beta

Posted 5 Oct, 2010 by Kord Campbell in Business and Startup

The extended private beta version of Loggly was pushed to production last Friday, October 1st, 2010. This release was planned in early August for release by end-of-September, so we did pretty well on getting it out when we did.

We called this release ‘the extended private beta’ because, well, we are extending the private beta to include more people who signed up on the beta registration page. We were so busy writing code for the release we couldn’t think of anything more exciting than this. Shoot, we were so busy I actually forgot to blog about the release!

If you haven’t gotten your invite yet, please be patient. We’re launching servers as fast as we can! Here’s a screen shot of the thing to tide you over:

Although we’re still in private beta, our roadmap for getting public beta is only a few months out. We’re planning on rolling out paid services for the private beta users toward the middle of November, and we should open up access to all registrations by the middle of December.

Min Product – Max Volume

In these days of rapid minimal viable product launches, we’ve been comparatively slow in launching Loggly’s service. Unlike other MVP offerings, Loggly will be expected able to handle customers that send in anywhere from a few MB to multiple GB of data per day of log files. Before we launch we have to test the systems will scale, and won’t melt under load. The extended private beta is part of that testing.

While we think we’ve got the scale issue licked for the moment, we’ve decided to add a feature which rate limits early beta accounts to 200MB/day of data. So we don’t block the upstream syslog servers, we’ll continue to accept and count event data, but will discard any data which exceeds the account limits. Limits are reset at midnight GMT, but if you need a higher daily limit please let us know.

Over the next few months we will raise these limits for the private beta users, and when we launch public beta sometime in December we’ll have a freemium account which will do somewhere in the neighborhood of 250-500MB/day. Paid tiers will be created to handle volumes above these rates. Paid tiers will also have additional features available to them such as S3 storage access, Hadoop processing, etc.

This release was the first of many steps toward making Loggly kick some serious ass when it comes to storing and searching the log files coming out of your infrastructure.

If you are interested in helping us beta test the product, please fill out the signup form and then give us a shout out on Twitter. We’ll see what we can do to get you on ASAP!

No Comments   |   Leave a Comment   |   Tweet

We Got Funded Again

Posted 18 May, 2010 by Kord Campbell in Business and Startup

On Monday, Loggly closed a $4.2M B round, with Trinity Ventures leading and True Ventures participating. As you may recall from my previous post, True led our initial seed investment, which was closed 5 months ago to the day.

My relationship with Trinity goes back over a year and a half ago – well before Raffy and I started thinking about doing a cloud based log management offering. Like many other startups, Trinity was started by two entrepreneurs. For years, Trinity’s motto has been focusing on early stage companies in specific technology categories, such as cloud computing and systems management.

Loggly is extremely fortunate to be working with Trinity and their bright team, and we greatly value the market experience they bring to the relationship.

History Lessons

I met Trinity through a fairly short introduction path. My good friend and former colleague, Dakota Sullivan, introduced me to a gentleman named Matt Strand in January of 2009. Matt and I had coffee at Crossroads Cafe in South Beach where I told him I was looking to join or start a cloud computing based company. Matt figured he should hook me up with a VC buddy of his, Dan Scholnick at Trinity.

Here’s the email introducing the two of us:

Matt was right about it being a valuable connection. Over the next year Dan and I would spend time together drinking coffee, chatting on the phone, and emailing each other about ideas in and around the enterprise and cloud computing space.

It was because of my conversations with Dan that Raffy and I were able to come to the idea of a cloud based logging service. Even when it came time to start pitching Loggly to others, Dan and Noel assisted us in honing our pitch, which eventually led to us being funded by True in a seed round.

Start Small, Go Big

When you are starting out, even the smallest conversation or the shortest email could potentially be the most important one you’ve had in years. Having an idea, growing it, and turning it into a business is a complicated process. That process takes time, and doesn’t happen over a matter of days, or even weeks, but instead over months and even years.

Our relationship with Trinity has been a long time in the works. While it may have appeared to happen rather quickly, Loggly’s efforts with Trinity started at the very beginning of its life.

In as much as your idea should evolve over time, your ability to convey the idea and the opportunity it represents should grow as well. I’ve lost count of how many times Dan has told me to ‘crisp up’ my presentation, discussed with me partnership negotiation strategies, or told me how to approach feedback with our current private beta testers, but I’m sure the hell glad he did.

Without investors like Trinity and True, it’s unlikely I’d be here telling you this story. You would be well to seek out these types of investors when you are looking for direction and guidance for your idea.

Now you’ll excuse me while we get back to coding. We have beta testers who have logs in need of indexing!

7 Comments   |   Leave a Comment   |   Tweet

We Got Funded

Posted 17 Dec, 2009 by Kord Campbell in Business and Startup

Loggly just closed an A round with True Ventures on Wednesday. From start to finish, Raffy, Jon and I talked to over 20 capital firms, with fund sizes ranging from a few hundred thousand dollars to over a billion dollars invested. In all, we spent exactly 90 days on our capital raising efforts, starting with essentially nothing, and then authoring and tweaking the executive summary, financial model, and investor presentation as we went. Oh, and we wrote a crapload of code in there too. The Loggly Beta deadline waits on no man.

Perhaps it was fate that we spoke with Puneet Agarwal at True Ventures first. True has a massive amount of experience investing in and managing early stage companies. Their record of past successes speaks for itself, and their team has experience with over 100 early stage investments that have generated significant investment returns. Frankly, Raffy, Jon and I are extremely fortunate to be working with the True Ventures team.

That first meeting with Puneet was actually quite easy; we had no other expectations other than sharing what we were thinking with someone who knew the space well. That was the calm before the storm though – over the coming weeks we struggled with writing our investor deck, meeting schedules, market size expectations, investor lack of familiarity with our market, and became consumed with correctly casting the “going big” portion of our pitch.

Going Big

The best way to describe what “going big” means is to just be blunt about it. It means, “How are you going to make your idea – your startup – compete effectively in a multi-billion dollar market?” You know, how are you going to get big like Apple, or SalesForce, for example. “Say what?”

A bunch of you capital guys and seasoned entrepreneurs will nod your heads vigorously at this statement. “Yes, yes. You need to show how this gets really big!” And, for all practical purposes, you guys are absolutely right. For a largish VC, say with a fund of a billion or so dollars to invest, they HAVE to go with early startup guys who are going to go really, REALLY big. It’s not a matter of money, it’s actually a matter of time. If you have a BILLION dollars, and you invest a million in each company you fund, that’s a THOUSAND companies you need to talk to, investigate, vet, poke at, wrangle with, grow to love, etc. Yeah, no. That’s not going to work unless you focus on a given market.

You’d have to filter fast. Kick out the guys that MIGHT do a run rate of low 10s of millions a year (crazy, right?) because they aren’t big enough. Shoot for the guys who tell a good story about how they are going to turn into Twitter, or Facebook, and exit for billions. Find those guys! This results in an investor funding a bare handful of early stage startup each year, even if they say otherwise on their website.

It also serves another purpose, all those meetings with those small startups. It allows an investor to form early relationships with companies who are successful at getting through the valley of death. If an investor finds out a startup they talked to early on is doing well, has revenue coming in, is growing, and expanding to the “going big” event, then maybe they might need some more capital. Maybe it’s time to invest.

Entrepreneur Up

If you are doing an early stage startup and are going to raise capital, you need to toughen up a bit before you go out. Remember, it only takes one firm to believe in your idea, but you are going to get an inverse number of rejections before that event transpires. If you get a bad review from someone, take their advice in stride and figure out how it applies to you. Tweak as needed, and move on to the next firm to vet what you’ve discovered.

Above all, be honest with yourself and your assumptions and don’t give the investor who gave you a bad review a hard time. If you think you’ve been asked to prove something unreasonable, like how you are going to become a billion dollar company when it’s just the 2 of you and a 1,000 lines of code, then say as much to the investor. Don’t be afraid to say you don’t know, or restate what you do. Don’t be afraid to talk through how you get big with the investors.

After being asked how we got to a billion dollar valuation by one VC, I turned it around on him and asked how he knew he was going really big on his company (which exited for >$1B). His answer? “We didn’t know until we got there!”

Loggly is going to go big, of that I assure you. But first, we have a product to build, customer and partner relationships to forge, and problems to solve for storing a ridiculous amount of log files. Once we have these tasks behind us, we’ll have a great handle on how we’re going to go really, really big.

9 Comments   |   Leave a Comment   |   Tweet

Maximizing Conversions in a Freemium Webapp - Part One

Posted 19 Nov, 2009 by Kord Campbell in Business and Startup

We’ve been on the investor tour for the last month and a half at Loggly. We spent about a month working on the investor presentation, executive summary and revenue model to prepare. Based on the better feedback we’ve received, we’ve continued to refine the pitch and plan.

We’ve heard everything from “you need to show me how this is a billion dollar company” (lol wut?) to “move the team slide just below the problem statement”. Bad advice aside, I remain focused on our market size and, in particular, the user conversion assumptions that go along with it.

I’ll be transparent here. I expect Loggly’s model to yield a 10-20% conversion rate to paid customer from a signed up freemium account. While I have good justifications for those numbers, posts like this, by Derek Haynes of ScoutApp, would appear to contradict my assumptions, especially with comments like “The 1% rule is the Pi of freemium web apps!”. In all fairness he has an excellent point about focusing on retention optimization, but should we so blindly ignore our conversion pipe just because of the 1% rule says we’ll get the users?

Theories != Rules

As it turns out, the “1% rule” is based on observations on other websites and expected conversion from a site visitor to a paid user. Coincidentally enough, this assumption echos the controversial content creation theory and smacks of Chinese Math.

The main problem with this approach is it completely and utterly ignores the remainder of a given site’s user pipeline. Pipelines, or funnels, are the bread and butter of any sales driven organization. While sites like YouTube may have simple funnels, sites like Loggly are a bit more complicated, and a missed assumption somewhere along the way can turn your assumed 1% conversion into a .1% conversion fairly quickly. To illustrate, here’s a short list of what we’ll be tracking at Loggly (per month):

• number of ad views on ad network


• number of visits


• uniques


• % visits and uniques from ads


• bounce rate


• content consumed (price sheets/videos)


• participation in the demo (low friction getting started)


• freemium signups


• use of a freemium account


• use of certain features in the freemium account


• conversion rate to paid accounts


• use of paid account


• logging rates


• upgrades from one paid tier to another

It’s within this pipeline that we focus our attentions to help maximize conversions from visit to paid user.  Not surprisingly, the devil is in the details.

Care and Feeding of Your Pipeline

Portions of a pipeline need to have estimated percentages assigned to them, and then you need to actively monitor those conversions from step to step when the site goes live. I’ll start with the expectation of 1 conversion from 100 unique visits to a paid account and then project assumptions about the other portions of the pipeline:

• 10,000 ad views yields 100 unique visits (1%)


• 100 unique visits yields 20 content consumers (20%)


• 20 content consumers yields 10 demo participators (50%)


• 10 demo participators yields 5 freemium signups (50%)


• 5 freemium signups yield 1 paid account (20%)

Notice this gives us our 1% conversion from unique visit to paid user, but still shows a 20% conversion from freemium to paid account. Now we do a breakdown for the intermediate steps, and find places we can optimize. Your mileage may vary, so make sure you understand your steps, and measure the effects of changes on your conversion numbers when you are able to do so!

Content is King

Based on prior experience, we know the subject material of Loggly should yield decent conversions from a visitor to content consumer, where the parties are interested in the offering. At previous engagements we were able to get 10K unique video views a month off 100K visits per month. Add in another 10% viewing pricing, examples, etc., and 20% starts to seem reasonable. By driving viewing content, we increase stickiness, educate the users about the service, and increase conversions to freemium signup.

Once the user is viewing content, we want to convert them to using the demo, and then on to creating an account later. Loggly’s demo will actually be a live demonstration of the service using the user’s own content, but it won’t require the user to sign up in the traditional sense. By flipping the signup process on it’s head, we get the user using the features first, then ask them for account information after they’ve decided they like the service. All in, and based on the frictionless signup, we expect 25% of users who viewed advanced content to convert to a freemium account.

Get Out of the Way!

Remember, your signup form is a HUGE barrier to users. If you require them to provide 12 fields of data to fill out, it’s that much less likely you’ll get a signup out of the deal. Just look at this signup form – I gave up after 5 minutes of trying to fill it out properly (picky password, clearing fields on submission, etc.), even though I’m highly interested in IBM’s offer. Most people won’t spend more than a minute or so filling out your signup form. The lesson here is to get out of the way so a user can start using your service quickly.

Finding Cash in the Pipe

Remember, freemium accounts are not the same as free trials. A free trial expires, but may contain all the features of a paid account. A freemium account doesn’t expire, and usually contains a subset of features of the paid account. By keeping the freemium account active for a longer period of time than the trial (and by keeping the cost of providing services to it low) you extend the amount of time a freemium user can convert to a paid user. While this may not show up in the first few months, over time it will have a compound effect on your conversions as users desire more features from the service.

At this point in the pipeline we are at a 5% conversion rate from visitor to freemium account, which so far seems to be a reasonable assumption. If we take the 10-20% estimated conversion rate to paid assumptions, then we arrive at a .5% to 1% conversion rate from visitor to paid account, which is within striking distance of our 1% goal.

In the next segment of this post, I’ll discuss analyzing how users use their accounts, and steps you can take to maximize conversion from freemium to paid.

Edit (11/12/09): Fixed my bullets in the second list.

4 Comments   |   Leave a Comment   |   Tweet

Three for the Win

Posted 5 Oct, 2009 by Kord Campbell in Business and Startup

I’m delighted to announce Jon Gifford as the new addition to Loggly’s founding partners. Jon will be taking on the roll of CTO at Loggly, and will acting as Chief Architect for the project. Jon hails from New Zealand, by way of Australia, and did stints at LookSmart, Technorati, Scout Labs, and recently his own startup, Minimal Loop. Jon is a search technology guru who is capable of accelerating search far beyond that of mere mortal men. If you don’t believe me, just try to go search for him on Google. Frankly, I have no clue how he does it. Trust me folks, a bunch of those those results are not him.

As for the rest of the team, Raffy is now officially our COO and Chief of Product (BTW, Swiss PMs rule), and I’ll be serving as our Geek CEO and Chief Evangelist. All three of us will be coding on Loggly over the next few months, and with a dash of a soon-to-be-announced designer, we’ll be assembling the framework needed for getting Loggly up and running in private beta over the coming months.

I’m really excited to be working with such smart and capable individuals. I can’t wait to start working with equally smart and capable partners and customers.

See you in your logs!

1 Comment   |   Leave a Comment   |   Tweet

Five Minutes of Fame

Posted 9 Sep, 2009 by Kord Campbell in Business and Startup

The first pass of the audio for the Loggly intro video was finished today. The clip will be used to further refine the wording of the script and start assembling the various visual elements we’ll need for shooting the video. What we should end up with is something similar to the works by Common Craft.

Not surprisingly, it took a TON of work to get the clip to its current state. Most of that work involved writing and editing the ‘script’ to sound like the actor was talking to an audience. I’m no script writer, so it took a fair amount of work with Brenda (the voice actor to whom I’m married) to nail down what sounded natural when spoken, and what didn’t. There are still some rough spots, but here’s the final version of the first draft if you want to take a listen. The script is here, less some final edits.

Back in the day, when John Leestma and I were involved in producing the Splunk Videos, we would do outtakes of the different developers doing their thing during filming. You had to do something to keep things entertaining – a 5 minute video would take all day to film and occupy 1/3 of the available conference rooms. It’s really too bad we didn’t publish them, some of them were side splitting.

Speaking of cracking up, the wife and I have always enjoyed a good laugh together. Here’s hoping you get a hoot from our silly outtakes. All your logs are belong to us!

No Comments   |   Leave a Comment   |   Tweet

Branding Yourself Blue

Posted 21 Aug, 2009 by Kord Campbell in Business and Startup

I bought a Baby Blue Bottle mic for my wife a few years back. She’s an opera singer, and can actually belt out some serious tunes when the mood strikes her. With her pipes and this mic we’ve gotten some amazing takes. If you’ve never seen/handled/heard a Blue mic, well, it’s a thing of joy to behold.

Now I’m working on Loggly fulltime, I realized we need a video which would explain the product in an easy to digest format, along the lines of the Twitter in Plain English video. It struck me I have all the bits I need to do this myself, including an amazing performer voice (albeit not using her operatic mode) and a kick ass microphone.

Videos are one of those things that help brand your identity, along with your blog posts, tweets, and of course, your product. The way in which you present your product to the people who use/consume it helps sear your identity forever in their minds. Case in point, Bluebottle Coffee. A friend of mine took me there a few weeks back and I was blown away by the preparation process. It was a thing of art really, much like the Blue Bottle Mic.

Maybe we should name ourselves Blue Bottle Logs. Whaddya think?

No Comments   |   Leave a Comment   |   Tweet

Loggly's Birth

Posted 17 Aug, 2009 by raffy@loggly.com in Business and Startup

Hello, we are Loggly, a startup located in San Francisco. As you can tell from our Web site, we are in the very early stages of our startup. Shoot, we haven’t even had a chance to let our designer take care of the Web site yet! So, tread with care – anything you find here is early stages!

1 Comment   |   Leave a Comment   |   Tweet