A few days ago an editor from CIO.com asked Loggly to provide two tips on hiring/retaining engineering talent for a future article. “Sure -- happy to help”, I replied. I then mentally outlined how we build and keep our highly sought after development team happy (out of
24 employees 25 employees, another just signed, 22 of our hires write code). The standard Silicon Valley startup checklist popped up:
- Competitive salary, equity packages, etc
- Open vacation, flex hours, etc
- Catered lunches, snack bars, and top-shelf beverages
- Latest-greatest technology and development environments
But to really attract, build and keep top talent you need more. You need the ability to:
- Work on a product that thrives on big data for a formidable challenge ($100b market, per IDC)
- Solve a true market created big problem, that matters... and one that people pay for
- Make a difference, and be recognized for it.
The last line was the true angle to build off and two examples quickly came to mind: how we target our hires and welcome them, and how we keep engineers aligned and motivated on the problem their code will solve, essentially the direct connection to our customer's log management challenges.
I’ll skip the tip I sent over on our on-boarding process, maybe I shouldn’t ever share it ;) but the second tip was about sending our developers to work a half-day in at the Loggly booth at a tradeshow. After reviewing our article suggestions, I heard back from the CIO.com editor:
"Thanks for the tips... Unfortunately, they weren't quite right for the article.
(Every IT guy I have ever known has DREADED being asked to man the booth or attend trade shows and conferences)."
I was shocked... and then I wasn’t. It only confirmed what I’ve already known, we are not our father's IT company. We aren’t:
- Building an old-school IT product (legacy deployment cycles aimed at non-cloud savvy companies run by stiff CIOs)
- Selling an old-school IT product (hundreds of end-users who are also our buyers sign up WEEKLY)
- Selling into the CIO and expecting our technology to be forced down (vs adopted, loved and shared)
- Looking, hiring or retaining old school ‘Dilbertized IT guy' engineering talent who hides at the mere thought of customer interaction
So why would I expect CIO.com to immediately see the value of the tips? I shouldn’t. At Loggly, we are different by design. We've developed a 100% cloud-based service loved and used by over 2,500 of the industries leading cloud-centric brands (AirBnb, Adroll, Sony, EA, GrubHub, Heroku) and we have a fun mascot named Hoover people love and want to interact with (here's Loggly's Hoover vs. the @Spotify shark at #Pycon).
This translates directly into my role driving Marketing and Revenue where I look for innovative developers and SysOps people (guys and gals) who are looking to simplify log management (we can help you!! Free 30 day trial of Loggly
), and it flows directly into who we hire, how we retain talent and how we motivate our teams.
Heading into PyCon
, I had the envious Marketing challenge of selecting booth staff after getting numerous emails, Skypes and SMS from engineers asking if they could attend and work the booth
, a great problem to have. As it turned out, our CTO and two lead developers attended for a half-day and it served to energize them about:
- The billion dollar opportunity we are solving in log management (the booth was busy!)
- The ability to hear from customers and prospects exactly what they want in a product and how they are using it (the good and the bad)
- The way they can make a difference with their efforts to improve and continually optimize the product (our customers enjoyed the face-to-face conversations)
Our engineers arrived at back at work on Monday really stoked to share their booth experience, their conversations, and on how they can make the next killer version of cloud-based log management to solve the challenges of our next 2,500+ customers.
So while the tip wasn’t good enough for CIO.com, I’ll take that as badge of honor. We are doing something different, things are going great
and people are excited to play a role.
If you aren't the luddite "IT-guy" and a person looking for a change, let's chat! Or please send a peer our way.
| Leave a Comment
Writing software is hard. Writing it well is even harder. I have been rolling some ideas around in my head on how to make software that is better for the people creating it and the people using it. Software has some distinct (and not-so-distinct) phases that it goes through as you work on it. The first of those phases for developers is the design phase. Here are some ideas on making better software during the design process.
If you start with "you know what would be cool?", kill the idea.
OK, so maybe that's a little harsh, but that's my knee jerk reaction when I hear a "you know what would be cool" idea. Let me rephrase my suggestion; stop and think. You may be wondering what I have against doing cool things. Nothing. There's nothing wrong with doing cool things. The problem is that we developers gravitate toward using a cool piece of technology for the sake of using it rather than finding the best fit for our problem. We often attempt to solve a problem either using a cool new piece of technology in a way that it wasn't intended for, or new language features when an alternative would have been better suited. Redis, mongodb, rabbitmq, Python, Ruby, Clojure, Scala, Node.js are all great technologies that meet various needs in their own ways. Sometimes they may be the best fit for your problem, but not always. Just think twice when you think you're about to do something "cool". Developing software is less about being cool and more about building something that works well.
Aim for simplicity.
Software developers have a knack for building and stacking and inheriting and composing until the original simple solution is unrecognizable in a mound of clever engineering. When you're thinking about the next piece of code that you're going to write, start simple. Don't build something in your mind that is more complex than it has to be. You should always use the simplest thing that works, except for when you shouldn't. Next point.
The simplest solution isn't always best.
Sometimes the simplest thing just won't cut it. Sometimes you need that really weird class hierarchy or a really tricky algorithm or something that really hurts your brain to even think about. Usually this is going to happen on the second iteration of a problem. You should still always start with the simplest thing. If that doesn't seem to hold up, go on to something more complicated, but keep the complexity to a minimum.
Don't build more than you need.
It's often tempting to try to build a solution that handles every contingency or meets the needs a user doesn't even know what they have yet. When you are tempted to design everything into your application, think twice. Don't design what you don't need. If you can come up with a good reason for designing it in, then go for it. Otherwise, don't bother. Build just what you need and no more.
I've seen so many developers (including myself) focus on getting something working now and neglect foreseeable issues with maintenance and deployment (which are the two major areas that this impacts). When you are thinking about the next piece of code that you are going to write, take the time to make sure you are focused on the long term solution. This may sound like it contradicts my previous idea of doing the simplest thing possible, but I think these ideas can co-exist. Start simple. Get something working. But don't plan on getting it "justworking"; plan on getting it working so that the next generation of developers that work on it will be happy you made the decisions you did rather than having them curse your name.
These aren't new thoughts. I've heard them stated by other experienced developers in various ways. And they're not immutable laws of nature. I think they're good guidelines, but the world won't come crashing down on your ears if you do something different. But I do believe that your experience of designing software will benefit from it. And as a result, your software will be better for you and your customer.
| Leave a Comment
We've been talking about doing a Loggly swag store for a while, and I'm excited to announce it's finally here! Now everybody everywhere has access to our snazzy swag. We have some rocking t-shirts, stickers and beer glasses available for purchase. I'm not just saying this because I work here, when I wear Loggly t-shirts out on weekends people compliment me on them.
We’ve always made a point to produce swag that has lasting appeal. All of us have gone to conferences and collected a bunch of stuff, only to take it back to our room, look at it, and leave it there. We are proud to say that people actually wear our swag out and about, track us down at events specifically to get their hands on it, and tweet at us to ask about how they can get it. We've even been offered money for the shirts off our backs, no joke. We are incredibly lucky that the allure of our brand continues beyond our product. Even folks who’ve never used our product, and never will, love the branding and culture we’ve created and want to be a part of it. The Loggly store was born from this enthusiasm and support, and we’re extremely grateful for it.
We’d like to give a special shout out to our fantastic illustrator Andre Jolicoeur who has done an outstanding job bringing Hoover the beaver and the Loggly brand to life with his creations. We also owe a great deal of praise to Shopify, whose service made it a breeze for us to get our store up and running. The Shopify Textmate bundle made development simple and faster than I had anticipated, and the entire launch process was demonstrated clearly through the Shopify interface.
To celebrate our launch we’re offering 20% off all merchandise this week, redeemable by using the discount code beaverlicious at checkout. Happy shopping!
| Leave a Comment
We’ve all dreamed of doing our job while sipping a margarita on the beach, Loggly and Scalr can get you there...well almost. We love things that scale and automate and all things cloud and that is why we partnered with Scalr. Scalr is an open source cloud management tool that brings automation to web applications. The cloud is all about scalability, growth, and making things easier to manage. Scarl gives you the ability to manage as many servers as you need on different cloud computing services and adjusts load capacity as you have spikes and valleys in your traffic. It’s like a system’s admin machine that never sleeps, doesn’t need energy drinks and coffee, so now your system’s admin can actually can have a life. Loggly comes in and takes care of all the logs, we store them, make them searchable, as well as providing features such as monitoring, troubleshooting and user analytics so nothing is lost and you can figure out what your users are doing. Sit back, relax and see what Scalr can do for you.
- DNS Management: automatically created and updated for you
- Fault tolerance: servers crash, at the worst times, now the problem is detected and automatically resolved for you
- Multi-cloud deployments: no need to commit to just one, infrastructure can be spread out across multiple providers
- Integrated SSH and key management
We're super excited to be partnered with such a cool company that's helping web apps scale and making your life easier.
| Leave a Comment
I recently came across an eWeek article titled 2012: A Cloudy Year for Big Data by Frank Ohlhorst. You could easily say I have a few opinions on the matter of big data! :)
First, I agree with Frank’s first notion that big data is neither big or new. The fact is, I've been saying things like "Dude, that's a ton of data!" since I started notching out the opposite sides of floppies back in the 80s. Remember these?
Ohlhprst quickly follows up his vague handwaving that ‘big data’ a new term with, “For most of its existence, big data has been out of the reach of small and midsize businesses (SMBs) because the storage and processing power needed to make this technology work is too expensive.”
Companies have been doing for years what they need to do better business, regardless of whether or not it’s expensive. In manufacturing, the costs of a small company optimizing on how to efficiently making tons of a cheap product can actually be quite a bit more expensive than a larger company making a few units of a complex product. In the same vein, smaller business may have more complex business optimization processes than larger ones, and require relatively larger amounts of data are required to solve those problems than with larger companies.
I agree with Frank that small business typically don't always have the resources necessary to solve massive scale problems, but again the problems are relative. For example, small software startups don't have project managers where larger ones do, not because they can’t afford them, but because they really don’t need them in a full-time capacity. I think this may be part of why SaaS services have been a huge hit and the term cloud has taken off because of it. SaaS allows companies to tackle a wide variety of problems across the entire business, all the while providing cost effective high tech solutions to solve problems in a way you could never have done before. For the first time in history the quality of business processes is experiencing sustained growth.
“These new cloud-based capabilities are on a growth path and are creating more opportunities for even the smallest of businesses to leverage big data without the traditional expenses of compute farms and massive storage arrays.”
Yes. However, compute farms and storage aren't the main thing that these companies need. They need access to the raw data that contains the data about their business, and the tools to extract the data in which they can take action. Figuring out your company’s problems requires brain power, understanding, data and tools. CPU and disks don't solve complex problems. People do.
It's All About Application Analytics
Ohlhprst also describes big data analytics as being comprised of three primary elements: volumes of unstructured data, processing power and algorithms. However, big data doesn't always imply unstructured data. Log files, or what Loggly calls Big Time Data™ typcially contain a large amount of structure. Dealing with structured data isn't always easy, and if you write software that 'expects' a certain structured format, your analysis can sometimes be broken or flawed if it encounters data that doesn't fit the structure you coded for. One way around this problem is to apply extra meta data to the data set. One technique to solving to this problem is adding a search index to the data, which is the approach Google pioniered and what Splunk and Loggly do for log files or time series event data. By being able to do text search data, and interact with it in realtime, or near real time, the user can optimize on solving the problem
Ohlhorst continues, “For it to be true big data, there has to be lots of it, and most SMBs don’t generate that volume of data internally, which leads them to seek out alternative data sources. Here, the cloud delivers.” Not true. Big data should be defined as an amount of data that a human can not reasonably digest. Generating large amounts of meaningful data is actually a bigger problem. Again, it's understanding the problem you have before you can solve it.
Yup. Ohlhorst explains that throughout 2012, data sets and others can be expected to grow exponentially - “The amount of data being generated globally increases by 40 percent a year", according to the McKinsey Global Institute, a data analytics research firm. True. The access of this data, mostly through the web, generates vast amounts of data as well. Ohlhorst continues that information needs to be organized, sorted and processed- and that takes computing power. Frankly nowadays, CPU is cheap enough that most of these problems can be solved on your laptop. Fast CPUs for crunching 'big data' aren't the problem any more than a search engine's main problem is crawling for data. The real bottleneck is adding meaning to the data that a customer can digest and make actionable.
PaaS/IaaS Accessibility Is a Problem
I’m glad that Ohlhprst recognized that Amazon isn’t the only one in the game in offering private cloud-based big data analytics platforms. He believes that since this technology is designed as a complete platform and not as a service, these platforms are still out of the reach of the SMB market.
Ohlhorst is right that these platforms are out of reach - but not just because they are designed as a complete platform and not as a service. I think it's because SMBs don't know they need it, don't have the data to put on it, and don't have the resources to manage it. There are plenty of hosted solutions out there (see SalesForce and their app marketplace) that provide some serious horsepower to the most important task - managing a company's contacts.
And of course, there had to be a Splunk mention in his article. Splunk sells expensive enterprise software. Their software is often times the most expensive piece of software a company has ever bought. Sounds like Oracle, eh? They aren't converting big data analytics into cloud services; they are simply taking their product and making a slimmed down version into a cloud offering they can generate leads with. Any serious big data customer they land will have to buy that very expensive solution and install it on a bank of computers and then pay people to manage it. SaaS is not what Splunk is taking to the market when they go IPO. It's their hellaciously expensive software licenses.
Big Time Data™. It's in the future of your small business.
| Leave a Comment
A few months ago I made an off-the-cuff remark about Loggly. "We're like one of those shitty solar powered calculators. When it gets dark, we forget everything you've typed into us."
That comment wasn't far off the mark. Historically, we haven't provided a whole hell of a lot of features that makes it easy to jump back into where you left off on your last search session. Basically when you logged out of Loggly, or even closed the shell in your browser, we'd forget everything you searched for until that point. It made it extremely difficult to get back to something meaningful the next time you logged in.
We shouldn't be here if we aren't meaningful. We should deliver users a 'punch in the gut' feature that makes a lasting impact. One they don't want to avoid.
Saving Time with Sticky Features
Scaling search for a massive amount of log file data being sent in from thousands of machines has been an overwhelming non-trival problem to solve for us over the past year, and it's been our top priority. Unfortunately us solving scale issues aren't readily obvious to users. Users always expect things on the web to be fast. They could care less how hard a problem it was to solve. They don't think to themselves, "Wow, that's fucking fast!". No. Instead they sit around and mutter things to themselves like "Why the hell doesn't feature X do Y? This thing is wasting my time!".
And there it is laid bare: Don't waste your user's time. It's the most valuable resource they have. Get to the point quick, make it easy to get back to what you were doing next time, and do it all with little fuss and muss.
I say give them sticky features!
Saved Search and More
And so, without further ado, I'm officially announcing one of many-to-come new sticky features: saved search. Saved search provides users a way to write a search query and then preserve the search to run again later. Saved searches can generate facet graphs or they can simply run a regular search across a given time range.
The shell has been reworked to provide context changes to use with the saved search feature. You can now change the date context, or limit the context to certain inputs, and then rerun the search or graph using the red rerun button at the top.
Here's a quick screencast running through some of our new sticky features:
Coming Up Soon
We're continuing to add features that increase stickiness to the product. Next week we'll be releasing a revamped history feature for the shell page, where what you've typed in before in a session will be preserved in your command history, just like it would in a normal shell prompt. We're also adding customized graph selection on the main dashboard, which will allow you to start viewing events that matter most to you by default when you first log in.
All these featuers are leading up to a major revamp of the way we provide value for our user's events. Expect completely customized dashboards for server monitoring, website performance, user analytics, and more soon! If you have a feature you'd like to see us implement, please do drop us a line. We're keen on not wasting your time!
| Leave a Comment
Sometimes there's just no other way to say "we're down" than just admitting you screwed up and are down. We're in the process of rebuilding the indexes of historic data of our paid customers. This is our largest outage to date, and I'm not at all proud of it.
So What Happened?
Yesterday afternoon all of our machines on Amazon's East region, availability zone 1d, were rebooted by AWS staff for maintenance purposes.
The cause of our failure is what some of you on Twitter are calling "a failure to architect for the cloud". I would refine that a bit to say "a failure to architect for a bunch of guys randomly rebooting 100% of your boxes". We've been told by Amazon they actually had to work hard at rebooting a few of our instances, and one scrappy little box actually survived their reboot wrath.
While some might go on a rant about how 'normal' failures don't affect 100% of your boxes the truth is that any and everything (including an army of reboot monkeys) can be expected to happen to your servers if you wait around long enough. The trick to being good at running a reliable service is to architect around any number of everythings that could happen to your service and build for it.
In this case we didn't build the workaround simply because the system we run - a combination of 0MQ+Solr+Zookeeper+Loggly Special Sauce - makes it extremely challenging to survive a complete failure with more than 1/2 of the cluster missing. With other challenges facing us, we decided to live with the risk.
So, How Do We Make This Right?
Single instances of Loggly's search cluster can't be spread across multiple availability zones or regions due to the amount of data we push around, latencies between the search nodes, and the lack of support in our system for redundant indexes. We've been OK with those limitations in the past simple because we systematically archive data to S3 when it arrives and we are capable of rebuilding indexes on the fly if we lose one or more indexers.
Our primary method to address this will be to start sharding our customers across multiple Loggly deployments. This will prevent further outages to the entire customer base. We've already been investigating other data centers on both dedicated hardware and other cloud-based services.
Finally, we accept full responsibility for the impact to our customers. We will be in touch with our paid customers sometime over the next week to address compensation for this outage.
We welcome feedback below.
Kord Campbell, CEO
| Leave a Comment
Writing software is tough, tools and processes that reduce the difficulty are always welcome. Code Coverage is an interesting developer tool because it tells you how much you don't know. Not having a test for a particular piece of code doesn't mean that it doesn't work, it just means you aren't as sure as you could be that it works. In this article, I will be covering getting the open source, Java Code Coverage tool, Cobertura, working with Ant, Jenkins and Github.
Since getting continuous integration working in a particular language can be complicated, it is a best practice to break the problem down into discreet chunks. Fortunately, Cobertura makes this easy, because the source code comes with an example ant project. You can download a binary distribution here: http://cobertura.sourceforge.net/download.html
. Inside of the download will be a relative path ..examples/basic
. If you cd into that directory, you can generate a code coverage report on a sample Java project by typing:
If you don't have junit installed, you will get output like the below, on my OS X Lion laptop:
If you run ant clean it will remove the reports directory, which contains, old reports. One tricky bit with running the example, is that the build.xml file is set to look for cobertura, and the cobertura lib directory, in a relative path two directories above.
This can be tricky if you check in only the examples directory into your git repository and don't have cobertura installed properly. What will happen then, is that the examples won't build properly and it will be tricky to figure out. If you find yourself in this situation, one easy hack is to simply place the cobertura jar files, including the dependencies in the lib, inside of your ant home. On Ubuntu linux this is /usr/share/ant/lib.
With the basic Cobertura configuration out of the way, the next thing to do is to create a throwaway git repository, on github or your own server. Next you will want to check in the whole Cobertura binary distribution that you download from their site: http://cobertura.sourceforge.net/
, or only check in the examples root, and install Cobertura properly on your build server. Inside of jenkins you will need to do the following things:
1. Point Jenkins at your git repository.
3. Configure the reports directory correctly. Because I only checked in the examples directory, my reports configuration looked like this:
4. build it.
Here is a screenshot of what it looks like when Jenkins builds the example project with code coverage:
If you were able to follow along at home, and have Jenkins automatically building the example project with code coverage, then the next step is to convert this knowledge to your own code base. One problem I had when getting Cobertura working with my code base was configuring the junit stanza properly to fork. Finally, an even simpler way to get code coverage cooking for your Java code base is to install the Eclipse plugin eCobertura. If you are really stuck getting things to work, this is a nice easy win. I hope you enjoyed the article, and if you have any Cobertura tricks, I would love to hear about them.
| Leave a Comment
One of the fundamental challenges of distributed coding is deciding what/when to integrate. Sure, that patch your colleague just sent you looks good, but is it actually ready to go into master? At Loggly, we've been feeling our way towards a disciplined integration process. A year ago, our frontend developers were all making commits directly to trunk in a single SVN repo. Once every few weeks, we'd run `svn up` on our servers, and hope for the best. Today our code goes through peer review, unit testing, and static analysis before it even touches our master branch.
Like most projects these days, the process starts on github. Fork. Push a feature branch to your repo. Open a pull request. Go through a couple rounds of discussion and revision. Merge. Every change to our code goes through this process. At first we thought it would slow us down, that we'd want pull requests for the nontrivial code and to just push to master for the easy stuff. After just a few days, we found the pull requests were slowing us down not at all, and that we all enjoyed the greater transparency into our colleagues' work.
Once we merge, the automation kicks in -- our default integration branch is 'proposed', so clicking merge doesn't actually get the code into the master branch. Jenkins polls our 'proposed' branch once a minute, then runs a simple preflight script on the code.
Rather than keep that preflight in a jenkins configuration page, we have it checked into the codebase so that any developer can run it too; this way there's no excuse for breaking the build -- you should have seen it break locally =P
Here's our preflight script. Let's go through it line by line.
DIR="$( cd "$( dirname "$0" )" && pwd )"
First we figure out where we're running, so that we can find the other scripts distributed with the app.
Next we purge pyc files. This is done because if a user recently switched from a branch which contained files which don't exist in our branch, the pyc files may still be around, and may be found by the interpreter.
Next we run a script to sync our python virtual environment, and ensure all requirements are present.
Here, of course, we run our unit tests. Each run prints a coverage report, so that as we recover from our testing debt, we can measure our progress.
Next, and this is important, we run pylint over the parts of our app that we expect to pass with no warnings. As we clean up our app, we continue to add modules to this list. Pylint does a few useful things for us. It looks for trivial name errors of the kind that could quickly cause code to stacktrace -- using a module without importing it, etc. It also enforces certain kinds of coding discipline. Our functions and modules can't exceed a certain length. The cyclomatic complexity of our functions is limited.
If all of this passes successfully, Jenkins automatically pushes the checked-out commit to master, which is where we base our development. Thus, we're always basing our development on known-vetted code.
If any of it fails, Jenkins still has a couple more tricks to pull. Here's our on-failure script:
This comes in two parts. The first runs a standard-issue git-bisect between origin/proposed and origin/master. Since origin/master has already been vetted by jenkins (that's how it became master), we know there'll be a regression somewhere between the commits. This goes into the session output, and is e-mailed to the relevant committers. Next, we roll the proposed branch back to the already-vetted master branch. Whatever pull request broke the build will have to be re-made from scratch.
| Leave a Comment
It gets real with the web in a major way. Websockets! I'm the new kid on the Loggly block and I've been working on Loggly's first alerting
app, Alert Birds
. Alert Birds uses Pusher because it's fast (real time is kinda hard to beat) and it's easy to use.
I'm giving you guys some code to play around with. Using Pusher and Sound Manager
, we're going to set up a site that makes Loggly's Alert Birds hop and sing on a page whenever anyone clicks on them. That means you see when your buddies click birds, and vise verse. Just think of it as a bird chatroom. Trust me, it's awesome.
If you want to follow along with the code, clone the github repo I created here:
All you need to do in order to get pusher working is sign up for an account, grab some credentials and get coding.
Pusher has two main components:
- Listening for events.
- Triggering events.
Let's start with listening:
To listen for a pusher event you only need the app key, channel name and event name. With these compoenets listening to a Pusher channel is pretty straight forward and looks something like this:
Triggering is just as straight forward and requires the same elements as listening, plus the app secret and a backend to handle and send data to Pusher.
First you'll need some data to send, let's go nuts with ajax!
This brings us to the backend component of triggering events, so let's look at the trigger.php file.
The first thing you need to do is head over to pusher's Publisher Libraries and choose the backend language you heart the most.
Once you do that simply follow the Library authors instructions and go nuts! In my case I chose the generic php library to demonstrate this simple example. It's all fairly straight forward but I did run into a namespace issue with my host. To fix it I simply commented out the namespace in Pusher.php.
In trigger.php I add Pusher.php using php's require function, put my keys, secrets etc into variables and use Pusher.php's Pusher() class to validate. The last line triggers the event and grabs data from the button variable I sent to the URL using my ajax call.
I should point out that Pusher is somewhat flash dependent as well. For browsers that don't support websockets, they have a solution. Flash. Which means it's probably a good idea to handle flashblockers for any pusher app you create.
At this point you should have everything you need to communicate with our Alert Birds. Go nuts!
| Leave a Comment
Back during my ISP heydays in the late 90s, I stumbled across a project called Peep The Network Auralizer. Peep basically takes things like netflow events from remote clients and sends them to a audio enabled server. Network traffic sounds like running water. Pings sound like birds chirping. SSH logins like frog croaks. The result is something that sounds like one of those sleep machines from the 80s.
What was really interesting about Peep, besides the fact it could lull you to sleep, was it could notify you of events that were important in aggregate but not necessaraly by themselves. A ton of failed SSH logins became a chorus of servers crying for help. High network load became a rushing river.
Now I look back on it, my use of Peep was my first pass at alerting and monitoring complex events in my infrastructure. Today it's also a good excuse to build an app for Loggly which makes a bunch of stupid bird noises.
Alerting is the #1 Loggly customer requested feature, so much so in fact that I broke down a few months ago and wrote a post on doing alerting with Loggly and Pagerduty.
After I wrote the post one of our customers commented that it sucked he had to run his own server to execute the cron jobs which ran the searches on Loggly. He's right. Requiring your users to run yet more infrastructure just to use a core bit of your service isn't the best way to win favors. And, it certainly doesn't scale well.
On that note, I'm pleased as punch to announce Loggly is now shipping alerting via the Alert Birds service. You can follow the Alert Birds on Twitter too!
Using Alert Birds with Loggly
Using Alert Birds with your Loggly account is simple. Navigate to the Alert Birds website and then log into your Google Account. Once you are logged in, you'll be given a form that allows you to enter your Loggly's account 'subdomain'. Once you submit your subdomain, you'll be taken to Loggly to login to that account and grant access to the Alert Birds' app. You'll be taken back to the Alert Birds' site when you are done authing the app to Loggly.
Alertbirds allows you to create an unlimited number of alerts, each of which are comprised of saved searches run on Loggly and endpoints that get called when searches return above or below a given threshold. Alerts are run on a schedule you define, and the endpoints currently (only) support pushing events to PagerDuty's APIs for notification. Alert Birds also allows you to trigger sound events in a browser viewing the Alert Birds' website.
Here's a short video on using the Alert Birds' application (sorry for saying um so much):
For more detailed setup info, you can reference our wiki page on Alert Birds. If you'd like to trigger an alert for Kord on his blog, click here.
Apps for a Logging Platform
We've Open Sourced the Alert Birds project to help illustrate how easy it is to build a logging application on Loggly. Alert Birds was writtien to run on Google's AppEngine service, but we'll be launching other apps soon that run on other frameworks and services.
Keep an eye out for more applications and app support coming out of Loggly in the near future. We're working on apps like a 3D globe for visualizing website traffic, web analytics, application reporting, and more!
| Leave a Comment
We've had the pleasure of having 3 great interns, that were often referred to as Intern 1, 2 and 3 "working" here during the summer and here is what they had to say in their own words about the Loggly experience. We miss you already!
Josept Friedrich (Intern #1)
I've definitely learned a lot since telling Kord I wanted to be a VC. In an effort to save my soul from the business world I've enjoyed improving my VIM skills and dabbling in emacs. School is an experience in and of itself but my time here at Loggly has given it a run for its money. Things I will miss:
- The 5th floor expeditions
- The warm beautiful summer weather of San Francisco
- Getting lost in the tenderloin
- The lunch conversations
- The meetups
- Fresh air
Escaping the heat in Texas couldn't have gone smoother, interning with Loggly this summer has been an amazing experience that I'll never forget.
Richard Berwich (Intern #2)
What I did at Loggly?
- Built a self-funded 12x12x8 box maze in the CEO's office while he was away at a conference.
- Left presents in the old office after moving to the new office. Yes, presents - two of them.
- Bought cookies for the office, and did not lace them with sedatives. You're welcome.
- Had a few beers, as well as a few awesome mixed drinks.
- Built an awesome application, which will continue to be developed in the future (by me, hopefully).
What I learned at Loggly:
- Enough Python to be able to get in trouble.
- How to make coffee (thank you, one button Cuisinart), but still drink hot chocolate (thank you, hot water pot).
- How to use and abuse github and Google App Engine.
- "Draw." (a few seconds go by) "BANG!"
Thank you, Loggly, for providing an awesome internship. I can't wait to come back after I'm out of school!
Keving Wang (Intern #3)
Working at Loggly has been a ridiculous roller coaster ride. At times I feel like I am God of my work, other times I feel like I am being worked like a slaveeeeeeeeeeeeeeeeeeee. All jokes aside, it has been an absolute pleasure to work here, everyone is amazing. From all the daily cursing and rage coding that we all do, to the random tasks that we do as interns, I happened to learn a lot about html and JQuery amongst other life lessons like T-shirt folding and breaking apart desks. I was able to complete a couple of projects during this summer with the help of Loggly's developers. This was one hell of an internship, I’m sure I’ll be keeping in touch with all the guys here including Joe and Richard while I’m at a school. Loggly is a magical place where you find yourself………T-shirt folding, box making, binge eating, head-desking and last but not least rage coding. I look forward to coming back here when I graduate.
| Leave a Comment
Aren Sandersen, VP Operations for Bebo, came over today and had lunch with us. Afterwards, we sat down and chatted about how Bebo is changing their infrastructure, manages logs, and how they use Loggly to do debugging, alerting and operational troubleshooting with Loggly. You can view the video on your iPhone via broadband, or watch the mobile version as well.
You can follow Aren and Bebo on Twitter, or sign up for Bebo on their website.
| Leave a Comment
Everyone seems to be blogging about how their service has ben impacted by Amazon’s AWS outage, or whining about how Amazon sucks, or explaining why their service was architected so well that it didn’t impact them, or why you suck if you didn’t plan for this.
As Loggly is based entirely on AWS and was only minimally impacted during the first few hours of the start of the outage, I figured I’d share exactly how we managed to do what we did:
We run across multiple availability zones and don’t rely on EBS for anything other than backups of a few simple databases. Everything else is file based the EC2 instances and their drives are set up in a RAID-1 configuration for speed and slightly more reliablity. Our log streams are backed up to S3 every few minutes as they come into our proxies. We rely on RDS for the database for the user logins, which really ended up being the only thing affected.
I asked Jordan Sissel, Head of Ops and Senior Developer here at Loggly to describe exactly what happened when Nagios/Pagerduty went off a night before last. Here’s what he said:
I saw RDS (prod db) problems in the early morning just as the problems started, but by the time I started debugging it the problem went away. I was notified by pagerduty because beaveroil and some other checks were failing.
Otherwise we weren’t really impacted. We got lucky, I think. I kept my eye on service but it stayed happy during the AWS outages.
Worst case, it’s easy for us (assuming rightscale is functioning, which it wasn’t for some of the day) to migrate to different parts of EC2 due to our use of puppet and are lack of EBS usage (only our RDS uses EBS)
Planning for Failure
Jordan is right, we can pretty much do a Loggly deployment on any AWS region within 20-30 minutes. Because we use Zerigo for DNS, and because we keep short TTLs, we can switch out records and have them updated quick and redirect ALL our inbound and outbound traffic to the new deployment.
Of course that leaves the question about migrating data on our existing or failed indexers. Thankfully that’s not a huge issue for us because we can rebuild them using EMR from our S3 backups.
Before we launched Loggly’s public service, I mandated that Loggly must be able to rebuild failed indexers at will. The work required to support this ended up delaying our public launch by at least 45 days. Now if we lose a box or have to move to another region, we can rebuild any (or all) of our indexers in at most a few hours. In theory we could continue to index new data coming into the system and historical search impact to customers would be minimal.
As my friend Clay Loveless put it so elegantly, “We may rip the rug out from under your feet at any moment.” If you haven’t planned for disaster striking, then you should go back and reassess your infrastructure. Hopefully we’ve planned well for just such a disaster.
<knocks on wood>
| Leave a Comment
My old friend David Myton from Boxed Ice swung by the Loggly office the other day to say howdy. I sat down with him and did a quick Geek CEO video about bootstrapping, developing product, filling the sales pipe, and listening to him being wise-beyond-his-years about raising capital.
Server Density is now growing at 20% a month, enjoyes a super low churn, and just got to break-even over the weekend. While others are just dreaming about startups, or grinding away for the man, David is here living the dream.
David Myton, CEO of Boxed Ice from Hoover on Vimeo.
David shared with me that the company will be moving to the Bay Area in a few more months, once they grow a little more and get their work stuff sorted around. They’re more than welcome to squat with us when they do!
| Leave a Comment
The extended private beta version of Loggly was pushed to production last Friday, October 1st, 2010. This release was planned in early August for release by end-of-September, so we did pretty well on getting it out when we did.
We called this release ‘the extended private beta’ because, well, we are extending the private beta to include more people who signed up on the beta registration page. We were so busy writing code for the release we couldn’t think of anything more exciting than this. Shoot, we were so busy I actually forgot to blog about the release!
If you haven’t gotten your invite yet, please be patient. We’re launching servers as fast as we can! Here’s a screen shot of the thing to tide you over:
Although we’re still in private beta, our roadmap for getting public beta is only a few months out. We’re planning on rolling out paid services for the private beta users toward the middle of November, and we should open up access to all registrations by the middle of December.
Min Product – Max Volume
In these days of rapid minimal viable product launches, we’ve been comparatively slow in launching Loggly’s service. Unlike other MVP offerings, Loggly will be expected able to handle customers that send in anywhere from a few MB to multiple GB of data per day of log files. Before we launch we have to test the systems will scale, and won’t melt under load. The extended private beta is part of that testing.
While we think we’ve got the scale issue licked for the moment, we’ve decided to add a feature which rate limits early beta accounts to 200MB/day of data. So we don’t block the upstream syslog servers, we’ll continue to accept and count event data, but will discard any data which exceeds the account limits. Limits are reset at midnight GMT, but if you need a higher daily limit please let us know.
Over the next few months we will raise these limits for the private beta users, and when we launch public beta sometime in December we’ll have a freemium account which will do somewhere in the neighborhood of 250-500MB/day. Paid tiers will be created to handle volumes above these rates. Paid tiers will also have additional features available to them such as S3 storage access, Hadoop processing, etc.
This release was the first of many steps toward making Loggly kick some serious ass when it comes to storing and searching the log files coming out of your infrastructure.
If you are interested in helping us beta test the product, please fill out the signup form and then give us a shout out on Twitter. We’ll see what we can do to get you on ASAP!
| Leave a Comment
On Monday, Loggly closed a $4.2M B round, with Trinity Ventures leading and True Ventures participating. As you may recall from my previous post, True led our initial seed investment, which was closed 5 months ago to the day.
My relationship with Trinity goes back over a year and a half ago – well before Raffy and I started thinking about doing a cloud based log management offering. Like many other startups, Trinity was started by two entrepreneurs. For years, Trinity’s motto has been focusing on early stage companies in specific technology categories, such as cloud computing and systems management.
Loggly is extremely fortunate to be working with Trinity and their bright team, and we greatly value the market experience they bring to the relationship.
I met Trinity through a fairly short introduction path. My good friend and former colleague, Dakota Sullivan, introduced me to a gentleman named Matt Strand in January of 2009. Matt and I had coffee at Crossroads Cafe in South Beach where I told him I was looking to join or start a cloud computing based company. Matt figured he should hook me up with a VC buddy of his, Dan Scholnick at Trinity.
Here’s the email introducing the two of us:
Dan <> Kord
Dan, please meet Kord Campbell. He is a serial entrepreneur interested in cloud computing, systems management, etc. with a few interesting ideas brewing. He is the tallest person I’ve met in at least a year or two.
Kord, please meet Dan Scholnick. He was one of the first employees at Wily and is now focusing on investments in your area of expertise for Trinity Ventures.
I think it’d be valuable for you two to connect. Let me know if there’s anything further I can provide, otherwise I’ll step back here and let you guys connect directly.
Matt was right about it being a valuable connection. Over the next year Dan and I would spend time together drinking coffee, chatting on the phone, and emailing each other about ideas in and around the enterprise and cloud computing space.
It was because of my conversations with Dan that Raffy and I were able to come to the idea of a cloud based logging service. Even when it came time to start pitching Loggly to others, Dan and Noel assisted us in honing our pitch, which eventually led to us being funded by True in a seed round.
Start Small, Go Big
When you are starting out, even the smallest conversation or the shortest email could potentially be the most important one you’ve had in years. Having an idea, growing it, and turning it into a business is a complicated process. That process takes time, and doesn’t happen over a matter of days, or even weeks, but instead over months and even years.
Our relationship with Trinity has been a long time in the works. While it may have appeared to happen rather quickly, Loggly’s efforts with Trinity started at the very beginning of its life.
In as much as your idea should evolve over time, your ability to convey the idea and the opportunity it represents should grow as well. I’ve lost count of how many times Dan has told me to ‘crisp up’ my presentation, discussed with me partnership negotiation strategies, or told me how to approach feedback with our current private beta testers, but I’m sure the hell glad he did.
Without investors like Trinity and True, it’s unlikely I’d be here telling you this story. You would be well to seek out these types of investors when you are looking for direction and guidance for your idea.
Now you’ll excuse me while we get back to coding. We have beta testers who have logs in need of indexing!
| Leave a Comment
Loggly just closed an A round with True Ventures on Wednesday. From start to finish, Raffy, Jon and I talked to over 20 capital firms, with fund sizes ranging from a few hundred thousand dollars to over a billion dollars invested. In all, we spent exactly 90 days on our capital raising efforts, starting with essentially nothing, and then authoring and tweaking the executive summary, financial model, and investor presentation as we went. Oh, and we wrote a crapload of code in there too. The Loggly Beta deadline waits on no man.
Perhaps it was fate that we spoke with Puneet Agarwal at True Ventures first. True has a massive amount of experience investing in and managing early stage companies. Their record of past successes speaks for itself, and their team has experience with over 100 early stage investments that have generated significant investment returns. Frankly, Raffy, Jon and I are extremely fortunate to be working with the True Ventures team.
That first meeting with Puneet was actually quite easy; we had no other expectations other than sharing what we were thinking with someone who knew the space well. That was the calm before the storm though – over the coming weeks we struggled with writing our investor deck, meeting schedules, market size expectations, investor lack of familiarity with our market, and became consumed with correctly casting the “going big” portion of our pitch.
The best way to describe what “going big” means is to just be blunt about it. It means, “How are you going to make your idea – your startup – compete effectively in a multi-billion dollar market?” You know, how are you going to get big like Apple, or SalesForce, for example. “Say what?”
A bunch of you capital guys and seasoned entrepreneurs will nod your heads vigorously at this statement. “Yes, yes. You need to show how this gets really big!” And, for all practical purposes, you guys are absolutely right. For a largish VC, say with a fund of a billion or so dollars to invest, they HAVE to go with early startup guys who are going to go really, REALLY big. It’s not a matter of money, it’s actually a matter of time. If you have a BILLION dollars, and you invest a million in each company you fund, that’s a THOUSAND companies you need to talk to, investigate, vet, poke at, wrangle with, grow to love, etc. Yeah, no. That’s not going to work unless you focus on a given market.
You’d have to filter fast. Kick out the guys that MIGHT do a run rate of low 10s of millions a year (crazy, right?) because they aren’t big enough. Shoot for the guys who tell a good story about how they are going to turn into Twitter, or Facebook, and exit for billions. Find those guys! This results in an investor funding a bare handful of early stage startup each year, even if they say otherwise on their website.
It also serves another purpose, all those meetings with those small startups. It allows an investor to form early relationships with companies who are successful at getting through the valley of death. If an investor finds out a startup they talked to early on is doing well, has revenue coming in, is growing, and expanding to the “going big” event, then maybe they might need some more capital. Maybe it’s time to invest.
If you are doing an early stage startup and are going to raise capital, you need to toughen up a bit before you go out. Remember, it only takes one firm to believe in your idea, but you are going to get an inverse number of rejections before that event transpires. If you get a bad review from someone, take their advice in stride and figure out how it applies to you. Tweak as needed, and move on to the next firm to vet what you’ve discovered.
Above all, be honest with yourself and your assumptions and don’t give the investor who gave you a bad review a hard time. If you think you’ve been asked to prove something unreasonable, like how you are going to become a billion dollar company when it’s just the 2 of you and a 1,000 lines of code, then say as much to the investor. Don’t be afraid to say you don’t know, or restate what you do. Don’t be afraid to talk through how you get big with the investors.
After being asked how we got to a billion dollar valuation by one VC, I turned it around on him and asked how he knew he was going really big on his company (which exited for >$1B). His answer? “We didn’t know until we got there!”
Loggly is going to go big, of that I assure you. But first, we have a product to build, customer and partner relationships to forge, and problems to solve for storing a ridiculous amount of log files. Once we have these tasks behind us, we’ll have a great handle on how we’re going to go really, really big.
| Leave a Comment
We’ve been on the investor tour for the last month and a half at Loggly. We spent about a month working on the investor presentation, executive summary and revenue model to prepare. Based on the better feedback we’ve received, we’ve continued to refine the pitch and plan.
We’ve heard everything from “you need to show me how this is a billion dollar company” (lol wut?) to “move the team slide just below the problem statement”. Bad advice aside, I remain focused on our market size and, in particular, the user conversion assumptions that go along with it.
I’ll be transparent here. I expect Loggly’s model to yield a 10-20% conversion rate to paid customer from a signed up freemium account. While I have good justifications for those numbers, posts like this, by Derek Haynes of ScoutApp, would appear to contradict my assumptions, especially with comments like “The 1% rule is the Pi of freemium web apps!”. In all fairness he has an excellent point about focusing on retention optimization, but should we so blindly ignore our conversion pipe just because of the 1% rule says we’ll get the users?
Theories != Rules
As it turns out, the “1% rule” is based on observations on other websites and expected conversion from a site visitor to a paid user. Coincidentally enough, this assumption echos the controversial content creation theory and smacks of Chinese Math.
The main problem with this approach is it completely and utterly ignores the remainder of a given site’s user pipeline. Pipelines, or funnels, are the bread and butter of any sales driven organization. While sites like YouTube may have simple funnels, sites like Loggly are a bit more complicated, and a missed assumption somewhere along the way can turn your assumed 1% conversion into a .1% conversion fairly quickly. To illustrate, here’s a short list of what we’ll be tracking at Loggly (per month):
- number of ad views on ad network
- number of visits
- % visits and uniques from ads
- bounce rate
- content consumed (price sheets/videos)
- participation in the demo (low friction getting started)
- freemium signups
- use of a freemium account
- use of certain features in the freemium account
- conversion rate to paid accounts
- use of paid account
- logging rates
- upgrades from one paid tier to another
It’s within this pipeline that we focus our attentions to help maximize conversions from visit to paid user. Not surprisingly, the devil is in the details.
Care and Feeding of Your Pipeline
Portions of a pipeline need to have estimated percentages assigned to them, and then you need to actively monitor those conversions from step to step when the site goes live. I’ll start with the expectation of 1 conversion from 100 unique visits to a paid account and then project assumptions about the other portions of the pipeline:
- 10,000 ad views yields 100 unique visits (1%)
- 100 unique visits yields 20 content consumers (20%)
- 20 content consumers yields 10 demo participators (50%)
- 10 demo participators yields 5 freemium signups (50%)
- 5 freemium signups yield 1 paid account (20%)
Notice this gives us our 1% conversion from unique visit to paid user, but still shows a 20% conversion from freemium to paid account. Now we do a breakdown for the intermediate steps, and find places we can optimize. Your mileage may vary, so make sure you understand your steps, and measure the effects of changes on your conversion numbers when you are able to do so!
Content is King
Based on prior experience, we know the subject material of Loggly should yield decent conversions from a visitor to content consumer, where the parties are interested in the offering. At previous engagements we were able to get 10K unique video views a month off 100K visits per month. Add in another 10% viewing pricing, examples, etc., and 20% starts to seem reasonable. By driving viewing content, we increase stickiness, educate the users about the service, and increase conversions to freemium signup.
Once the user is viewing content, we want to convert them to using the demo, and then on to creating an account later. Loggly’s demo will actually be a live demonstration of the service using the user’s own content, but it won’t require the user to sign up in the traditional sense. By flipping the signup process on it’s head, we get the user using the features first, then ask them for account information after they’ve decided they like the service. All in, and based on the frictionless signup, we expect 25% of users who viewed advanced content to convert to a freemium account.
Get Out of the Way!
Remember, your signup form is a HUGE barrier to users. If you require them to provide 12 fields of data to fill out, it’s that much less likely you’ll get a signup out of the deal. Just look at this signup form – I gave up after 5 minutes of trying to fill it out properly (picky password, clearing fields on submission, etc.), even though I’m highly interested in IBM’s offer. Most people won’t spend more than a minute or so filling out your signup form. The lesson here is to get out of the way so a user can start using your service quickly.
Finding Cash in the Pipe
Remember, freemium accounts are not the same as free trials. A free trial expires, but may contain all the features of a paid account. A freemium account doesn’t expire, and usually contains a subset of features of the paid account. By keeping the freemium account active for a longer period of time than the trial (and by keeping the cost of providing services to it low) you extend the amount of time a freemium user can convert to a paid user. While this may not show up in the first few months, over time it will have a compound effect on your conversions as users desire more features from the service.
At this point in the pipeline we are at a 5% conversion rate from visitor to freemium account, which so far seems to be a reasonable assumption. If we take the 10-20% estimated conversion rate to paid assumptions, then we arrive at a .5% to 1% conversion rate from visitor to paid account, which is within striking distance of our 1% goal.
In the next segment of this post, I’ll discuss analyzing how users use their accounts, and steps you can take to maximize conversion from freemium to paid.
Edit (11/12/09): Fixed my bullets in the second list.
| Leave a Comment
I’m delighted to announce Jon Gifford as the new addition to Loggly’s founding partners. Jon will be taking on the roll of CTO at Loggly, and will acting as Chief Architect for the project. Jon hails from New Zealand, by way of Australia, and did stints at LookSmart, Technorati, Scout Labs, and recently his own startup, Minimal Loop. Jon is a search technology guru who is capable of accelerating search far beyond that of mere mortal men. If you don’t believe me, just try to go search for him on Google. Frankly, I have no clue how he does it. Trust me folks, a bunch of those those results are not him.
As for the rest of the team, Raffy is now officially our COO and Chief of Product (BTW, Swiss PMs rule), and I’ll be serving as our Geek CEO and Chief Evangelist. All three of us will be coding on Loggly over the next few months, and with a dash of a soon-to-be-announced designer, we’ll be assembling the framework needed for getting Loggly up and running in private beta over the coming months.
I’m really excited to be working with such smart and capable individuals. I can’t wait to start working with equally smart and capable partners and customers.
See you in your logs!
| Leave a Comment
The first pass of the audio for the Loggly intro video was finished today. The clip will be used to further refine the wording of the script and start assembling the various visual elements we’ll need for shooting the video. What we should end up with is something similar to the works by Common Craft.
Not surprisingly, it took a TON of work to get the clip to its current state. Most of that work involved writing and editing the ‘script’ to sound like the actor was talking to an audience. I’m no script writer, so it took a fair amount of work with Brenda (the voice actor to whom I’m married) to nail down what sounded natural when spoken, and what didn’t. There are still some rough spots, but here’s the final version of the first draft if you want to take a listen. The script is here, less some final edits.
Back in the day, when John Leestma and I were involved in producing the Splunk Videos, we would do outtakes of the different developers doing their thing during filming. You had to do something to keep things entertaining – a 5 minute video would take all day to film and occupy 1/3 of the available conference rooms. It’s really too bad we didn’t publish them, some of them were side splitting.
Speaking of cracking up, the wife and I have always enjoyed a good laugh together. Here’s hoping you get a hoot from our silly outtakes. All your logs are belong to us!
| Leave a Comment
I bought a Baby Blue Bottle mic for my wife a few years back. She’s an opera singer, and can actually belt out some serious tunes when the mood strikes her. With her pipes and this mic we’ve gotten some amazing takes. If you’ve never seen/handled/heard a Blue mic, well, it’s a thing of joy to behold.
Now I’m working on Loggly fulltime, I realized we need a video which would explain the product in an easy to digest format, along the lines of the Twitter in Plain English video. It struck me I have all the bits I need to do this myself, including an amazing performer voice (albeit not using her operatic mode) and a kick ass microphone.
Videos are one of those things that help brand your identity, along with your blog posts, tweets, and of course, your product. The way in which you present your product to the people who use/consume it helps sear your identity forever in their minds. Case in point, Bluebottle Coffee. A friend of mine took me there a few weeks back and I was blown away by the preparation process. It was a thing of art really, much like the Blue Bottle Mic.
Maybe we should name ourselves Blue Bottle Logs. Whaddya think?
| Leave a Comment
Hello, we are Loggly, a startup located in San Francisco. As you can tell from our Web site, we are in the very early stages of our startup. Shoot, we haven’t even had a chance to let our designer take care of the Web site yet! So, tread with care – anything you find here is early stages!
| Leave a Comment