Loggly

At Loggly, we're not your father's IT company

Posted 1 Apr, 2013 by dave@loggly.com in Business, Log Management, and Startup

A few days ago an editor from CIO.com asked Loggly to provide two tips on hiring/retaining engineering talent for a future article.  “Sure -- happy to help”, I replied.  I then mentally outlined how we build and keep our highly sought after development team happy (out of 24 employees 25 employees, another just signed, 22 of our hires write code).  The standard Silicon Valley startup checklist popped up:

• Competitive salary, equity packages, etc
• Open vacation, flex hours, etc
• Catered lunches, snack bars, and top-shelf beverages
• Latest-greatest technology and development environments

• Work on a product that thrives on big data for a formidable challenge ($100b market, per IDC)
• Solve a true market created big problem, that matters... and one that people pay for
• Make a difference, and be recognized for it.

"Thanks for the tips... Unfortunately, they weren't quite right for the article.
(Every IT guy I have ever known has DREADED being asked to man the booth or attend trade shows and conferences)."

• Building an old-school IT product (legacy deployment cycles aimed at non-cloud savvy companies run by stiff CIOs)
• Selling an old-school IT product (hundreds of end-users who are also our buyers sign up WEEKLY)
• Selling into the CIO and expecting our technology to be forced down (vs adopted, loved and shared)
• Looking, hiring or retaining old school ‘Dilbertized IT guy' engineering talent who hides at the mere thought of customer interaction

• The billion dollar opportunity we are solving in log management (the booth was busy!)
• The ability to hear from customers and prospects exactly what they want in a product and how they are using it (the good and the bad)
• The way they can make a difference with their efforts to improve and continually optimize the product (our customers enjoyed the face-to-face conversations)

• Software Developer
• Sales Engineer
• Director of Product Marketing

No Comments   |   Leave a Comment   |   Tweet

If you love your logs, set them free

Posted 12 Mar, 2013 by Philip O'Toole in Business and Log Management

Imagine this. It's the night before Thanksgiving, you're having dinner with your wife and in-laws at a nice restaurant. Your cellphone rings. It's the Senior Director of Engineering, your boss, responsible for the new Cloud Portal you and your team brought up a few weeks back. This can't be good. And it happened to me, at a big well-respected engineering focused company. #truestory

It turns out a key customer had decided on this night, of all nights, to place a large order for licenses, but our Portal is returning HTTP 500. "What could be happening?", he pleads. "Hmmm, can we get the logs?" I reply. "We can't", he tells me, "Operations never set up systems to take them off the box." Sigh... Sound Familar?

This happens all the time within IT organizations. It turns out that one of the worst places logs can be stored is on the machine generating those logs. What if that machine contains Insider Information, and access to it is restricted? What if the network link to it is down? What if all you can do is think really, really hard to work out why that key customer can't place that order? But it doesn't have to be this way.

If you love your logs, set them free. Get them off the host machines. You, your boss, and your customers will thank you when you can diagnose issues from anywhere, and in real-time. Get them into Loggly, and let us do the hard work for you.  Best part, you can do it all without on-premise software, local agents, or disrupting production machines.

Anyway, my boss and I solved the issue, and the order was placed, but it took until the taxi ride home.  Life Lesson? Enjoy Thanksgiving and other holidays stress-free... learn from others so this doesn't happen to you.

No Comments   |   Leave a Comment   |   Tweet

The Truth is in the Logs

Posted 14 Feb, 2013 by dave@loggly.com in Business, Code, and Log Management

Last Friday, The New York Times reporter John Broder published a less than rosy picture of highway trip between Washington D.C. and Boston, cruising in Tesla’s Model S luxury sedan. The purpose of the trip was to range test the car between two new supercharging stations with a "speedy road trip". Broder wrote about his anxiety-ridden stretches between charging stations, when energy consumption was outpacing mileage. Among other complaints, he was unhappy about the need to power off the heat on a cold Northeastern day and to drive slowly, in an effort to conserve battery power.  The trip ended not at the charging station, but on the back of a tow truck having run out of electricity―a result he visually documented with a feature photo of the car being towed. Ouch.

This wasn’t the PR outcome that Tesla Motors chief executive Elon Musk expected. Avoiding a factless he-said-she-said on Twitter, he gave an interview saying that he was planning to publish the log files of the reporter's vehicle, since they were at odds with details in the story. As reported in Venture Beat:  “Musk claims Broder failed to mention how much he was punching the accelerator early in the ride, a move that Tesla warns its customers will drain the battery faster. Also, Musk says Broder took a detour through Manhattan. And he didn’t fully charge the car before departing.”

NYTimes article about Tesla range in cold is fake. Vehicle logs tell true story that he didn't actually charge to max & took a long detour.

— Elon Musk (@elonmusk) February 11, 2013

While the logs have yet to be published, if they are and if they do support Musk’s claims, it will be advantageous for Tesla -- and the “logging” community at large; bringing light to the next big category of business intelligence.  "Your honor, I'd like to call the #LogFile as my next witness." Amusing, but true and will be happening more and more often inside and outside of business.

As this story demonstrates, software is behind everything these days. Massive machine data is being generated every minute not only from our computers and cell phones, but from the cars we drive, the appliances we use, and virtually anything with a chip, motor or battery run with an operating system. The golden nuggets of truth exist within the log files, holding the detailed data about an event that happened, one that cannot be disputed.  

Such data, when mined and organized properly, provides a wealth of indicators to solve problems, such as why a web transaction timed out, the difference between a server being “on” and actually performing as intended, or to defend the veracity of a product’s claims. The tremendous value for IT departments (sysops, techops, devops and product developers) in having real-time access to log data for feedback on product performance is limitless.  Log files can show where users struggled or took too long to accomplish tasks, or where your applications or hardware let them down.  In trial-by-pubic cases like these log file data shines in a new light, far outside of the walls of technology but in the vernacular of the general public.

Logs don’t lie, but the truth can’t come out unless there are affordable and easy ways to release insights from these enormous log databases within the window of time in which they matter.  Moving the conversation forward, suppose Tesla was able to collect and analyze log files on all of its vehicles and then receive alerts on issues to determine if they need addressing or if they are simply isolated events caused by user error, right back to the driver before the situation arose.  That's the end game: taking aggregate user data to find nuggets of wisdom that can then be fed back to the end user with guidance -- seamlessly.

That's where we come in. Loggly’s cloud-based log management service gives companies fast, centralized access to all of their log data, so they can solve issues, identify problems and make customers happy again understanding and answering "is this needle in the haystack or tip of the iceberg".  In the court of public opinion and social everything or in the case of 100% cloud driven buisness... anything less than real-time is becoming really-late.

When consumer product CEOs start talking about logs, that's a pretty good sign that log files and log analytics are not just a stream of text and data to throw on the backup server every night―if you didn’t run out of storage for them already.  Smart log file mining helps companies keep customers happy and their bottom line bigger.  If you run a data-driven business, the more your company can act on that data to improve application/service/product performance and experience, the better off your customers will bef.

If Elon Musk is driving the future of Tesla, an automobile brand off his log files, shouldn't your cloud application-driven business also being harnessing the full power of log intelligence?

No Comments   |   Leave a Comment   |   Tweet

Uniq-uely Different

Posted 12 Jul, 2012 by Leila Tudury in Log Management

We've extolled the merits of sending in logs that are formatted in JSON for a while now, but I've been holding off on this post because I wanted to have something really exciting to tell you about. 

To bring the non-JSON users up to speed: you can format your log output so that it will send something like this:

   { 
     "Name": "Hoover Beaver",
     "Age": 30,
     "Occupation": "Tree Feller",
     "Address": "30 Hoover Dam Road"
   }

Instead of the typical unstructured log events:

"Hoover Beaver" 30 "Tree Feller" "30 Hoover Dam Road"

This means that you can search for "json.age:30" and know that you're only searching the Age field rather than the entire event.  Don't bother learning how to awk/sed, just tell us what field you want.

Now that everyone's on the same page, I can tell you about one of my favorite analytics features.  We've made some enhancements to our command "uniq".  "Uniq" works by collapsing duplicates and returning a word count.  For example, if I want to find out the distribution of twenty-somethings across all Tree Fellers, I can run the following command which will give me a count for every unique age between 20 and 29 across all Tree Fellers:

 > uniq json.age json.occupation:"tree feller" AND json.age:[20 TO 29]

count       json.age
__________________________
455             23
358             21
276             24
121             28

This also works well with strings.  If you're interested in seeing the distribution of error messages since your last big release, try something like this:

 > uniq json.error json.version:2.1

count       json.error
__________________________
146             ERROR 55: Trees are falling in the forest
102             WARNING 21: Too many beavers in the stream
89               ERROR 29: Can't find the right river
73               ERROR 98: The dam has broken

In the past, we used tokenized strings to report back uniq values, which made it quite difficult to filter out the useful information.  Today, we're storing fields < 100 characters as both tokenized & untokenized strings.  This analytics feature is available to only JSON users right now. If you need advice on how to change your log format & make the switch, email our support team.

Try it out and let us know how it goes!

edit:  If you want uniq values through our API, try using Facets.   Try something like this: 

https://SUBDOMAIN.loggly.com/api/facets/json.FIELD?q=inputname:INPUT%20json.FIELD:VALUE

Currently uniq search terms are limited to one within the UI.  If you'd like to use the uniq command for more than one term please use the API:

https://SUBDOMAIN.loggly.com/api/facets/json.FIELD/?q=json.FIELD:TERM1%20json.FIELD:TERM2%20json.FIELD:TERM3&from=NOW-7DAYS&until=NOW

No Comments   |   Leave a Comment   |   Tweet

Getting Your Product Sticky

Posted 13 Jan, 2012 by Kord Campbell in Log Management and Startup

A few months ago I made an off-the-cuff remark about Loggly. "We're like one of those shitty solar powered calculators.  When it gets dark, we forget everything you've typed into us."

That comment wasn't far off the mark.  Historically, we haven't provided a whole hell of a lot of features that makes it easy to jump back into where you left off on your last search session.  Basically when you logged out of Loggly, or even closed the shell in your browser, we'd forget everything you searched for until that point.  It made it extremely difficult to get back to something meaningful the next time you logged in.

We shouldn't be here if we aren't meaningful.  We should deliver users a 'punch in the gut' feature that makes a lasting impact.  One they don't want to avoid.

Saving Time with Sticky Features

Scaling search for a massive amount of log file data being sent in from thousands of machines has been an overwhelming non-trival problem to solve for us over the past year, and it's been our top priority.  Unfortunately us solving scale issues aren't readily obvious to users.  Users always expect things on the web to be fast.  They could care less how hard a problem it was to solve.  They don't think to themselves, "Wow, that's fucking fast!".  No.  Instead they sit around and mutter things to themselves like "Why the hell doesn't feature X do Y?  This thing is wasting my time!".

And there it is laid bare: Don't waste your user's time.  It's the most valuable resource they have.  Get to the point quick, make it easy to get back to what you were doing next time, and do it all with little fuss and muss.

I say give them sticky features!

Saved Search and More

And so, without further ado, I'm officially announcing one of many-to-come new sticky features: saved search.  Saved search provides users a way to write a search query and then preserve the search to run again later.  Saved searches can generate facet graphs or they can simply run a regular search across a given time range.

The shell has been reworked to provide context changes to use with the saved search feature.  You can now change the date context, or limit the context to certain inputs, and then rerun the search or graph using the red rerun button at the top.

Here's a quick screencast running through some of our new sticky features:

Coming Up Soon

We're continuing to add features that increase stickiness to the product.  Next week we'll be releasing a revamped history feature for the shell page, where what you've typed in before in a session will be preserved in your command history, just like it would in a normal shell prompt.  We're also adding customized graph selection on the main dashboard, which will allow you to start viewing events that matter most to you by default when you first log in.

All these featuers are leading up to a major revamp of the way we provide value for our user's events.  Expect completely customized dashboards for server monitoring, website performance, user analytics, and more soon!  If you have a feature you'd like to see us implement, please do drop us a line.  We're keen on not wasting your time!

No Comments   |   Leave a Comment   |   Tweet

Ajax This!

Posted 7 Nov, 2011 by Brian Schroeder in Code and Log Management

Today’s blog post is more of a tool than a toy.  Lately I’ve been working on a bookmarklet that utilizes Pusher (if you want to learn a little about Pusher, check out my previous blogpost).  The bookmarklet I’m working on is really silly, but these technologies have the potential to be used for some really cool apps.

I needed to figure out a way for javascript to manipulate objects on web pages remotely through ajax calls.   You can already do this with click events and Pusher by sending the class or ID of any element you click on, but what if there is no class or ID?  I also want this to work on any webpage.  As a result, I needed to figure out how to build a selector that was as unique as possible.  One way to do this is  to select using all attributes the object has to offer.

Enter, the custom attribute selector syntax:

This selector will only select objects that have a class value of “whatever” AND a myattribute value of “customattribute”.

Some stack overflow and basic google searching told me I can tease attributes out using a regex(gross).  I knew there had to be a better way so I started fiddling with javascript objects I captured on click and tried to figure out how to uniqify my selector string.  What I discovered is that every object I clicked on contained a ton of info.  The parts that I use to build my unique-ish selector are:

• localName

The name of the html tag.  in the gist above, this value is customhtml.

• attribute

All attribute(s) associated with the object (if any). In the gist above, these attributes are class=”whatever” and myattribute=”customattribute”

• parentNode

This is the object that contains the one clicked on.  Technically my gist has no parent, but I use the parent node to map out a path from the object I clicked on all the way up to the body of the page.  If the object you click on doesn’t have any attributes ( a <p> object for example) you can still select it based on the specific path to that object.

• textContent

This is any text that’s in the object.  For example <p>writing stuff</p> will have a textContent value of “writing stuff”.

Apart these elements aren’t bad at selecting the object you want, but together they get pretty close to behaving like a this object.  

Here is how I stitch these elements together to create my this-ish selector:

There are a few tricks here:

• I found that if the object you click on has no attributes the path doesn’t help too much, so if the object I’m building a selector for doesn’t have any attributes I just give it a class with an empty value, and it works really well.  
• In an attempt to keep my selector as unique as possible I’m also using :contains() to select on specific strings as well as the path.  This helps ensure that you’re manipulating the exact object you want.
• If you’re clicking on an image tag, I found that contains won’t allow you to select that image, but if we take it out it works, so I added a filter to keep image tag selectors from using :contains(). (although it currently only works in the developer tools javascript console, I haven’t yet figured out why it’s not working for my bookmarklet).

Generally, the resulting selector ends up looking something like this:

While this method doesn’t currently work on every single element of every single webpage, it has worked on most of the pages I have tried out, including the google search page.

Anyone who has every tried to explain, on the phone (so 1991), to a grandparent or non-tech savvy friend/family member how to do certain things on the web should be able to see the power of this.  With these selectors it’s now possible to build a bookmarklet that allows two or more people to see what elements are being clicking on.  Now describing webpage elements to your grandfather isn’t needed.  Hook him up with your Pusher powered bookmarklet and all he has to do is go to a URL, click a bookmark button, and let’s say... click on the text you just highlighted in yellow.  On the flip side, you’ll know (almost)exactly what elements on the page he’s clicking on because his clicks will make highlights on the page as well.  Super neat!

Obviously this method isn’t perfected, but if you’re interested helping me make it more precise, please fork my project on github.

https://github.com/brainTrain/ajaxThis

No Comments   |   Leave a Comment   |   Tweet

Java logging, and our awesome community

Posted 20 Sep, 2011 by jon@loggly.com in Code and Log Management

Since I'm on a roll with the blog posts, I thought I'd quickly cover some of the ways you can log out of your Java code to us. I've been buried in *our* java code for nearly two years now, and we've been talking about improving the quality of the logging that we're doing there recently. As it turns out, there are some pretty interesting projects out there that look like they may be just what we need...

Before talking about them, though, I thought I'd give you a quick run-down on how we're doing things now. We use a very slightly tweaked version of the log4j SyslogAppender  (version 1.2.16) - the tweak is that we upped the message size from 1k to 32k. Yep, we're completely ignoring the syslog RFC, but its been working perfectly fine for us for quite a while, so I'm ok with that. We configure log4j as described on our wiki so that we log to a local syslog-ng (using a different facility for each app) which forwards to the appropriate ports on logs.loggly.com. Its a very simple approach, but has been very reliable for us.

So why would we want to change it?

The main reason is that we set this up before we could send JSON data, and we log a lot of performance data, which is a perfect fit for JSON. We're going to be moving all of our java logging to JSON over the next few months, because it will let us dive deeper into our logs, without all the noise.

A couple of weeks ago, Patrick Lightbody from neustar emailed us to tell us about a java class he'd written to send data into Loggly using http. Its an extension of java.util.logging.Handler and is only a couple of hundred lines of nice clean code . He shared it on github and described how to use it in his email.

This got me thinking that we've been a little, um, remiss in communicating just how many libraries people have written for us, so I did a search on github for java projects with loggly in the repo name and found some other projects that also look pretty nice...

• Mårten Gustafson has implemented a logback/SLF4J appender
• Spidertracks has implemented a log4j appender that uses an embedded HSQL db to guarantee delivery of your log events
• Raoul Schmidigerhas is working on a log4j appender
• Thomas Engelschmidt is working on a clojure logger

All of these projects use our HTTP interface, rather than TCP or UDP, which makes sense since we don't currently support JSON except over HTTP. We're planning on fixing that :-)

Expanding the search a little, there are a bunch of projects in github, on top of the ones we've created, that should make it easy to log out of javascript, python, ruby, and C#, as well as the Java stuff I talked about above.

We're obviously pretty happy that so many people are working on making it easier to get data to us, and I want to say thank you to everyone who has been contributing to all of those projects.  I'd like to encourage all of you reading this to jump in and help out if you can. Everyone who is contributing to these projects should drop us a line, and we'll send you one of our X-Ray Beaver tee's as a thank you :-)

4 Comments   |   Leave a Comment   |   Tweet

Alerting with Alert Birds

Posted 13 Sep, 2011 by Kord Campbell in Log Management and Startup

Back during my ISP heydays in the late 90s, I stumbled across a project called Peep The Network Auralizer.  Peep basically takes things like netflow events from remote clients and sends them to a audio enabled server.  Network traffic sounds like running water.  Pings sound like birds chirping. SSH logins like frog croaks.  The result is something that sounds like one of those sleep machines from the 80s.

What was really interesting about Peep, besides the fact it could lull you to sleep, was it could notify you of events that were important in aggregate but not necessaraly by themselves.  A ton of failed SSH logins became a chorus of servers crying for help.  High network load became a rushing river.

Now I look back on it, my use of Peep was my first pass at alerting and monitoring complex events in my infrastructure.  Today it's also a good excuse to build an app for Loggly which makes a bunch of stupid bird noises.

Alerting Rules

Alerting is the #1 Loggly customer requested feature, so much so in fact that I broke down a few months ago and wrote a post on doing alerting with Loggly and Pagerduty.

After I wrote the post one of our customers commented that it sucked he had to run his own server to execute the cron jobs which ran the searches on Loggly.  He's right.  Requiring your users to run yet more infrastructure just to use a core bit of your service isn't the best way to win favors.  And, it certainly doesn't scale well.

On that note, I'm pleased as punch to announce Loggly is now shipping alerting via the Alert Birds service.  You can follow the Alert Birds on Twitter too!

Using Alert Birds with Loggly

Using Alert Birds with your Loggly account is simple.  Navigate to the Alert Birds website and then log into your Google Account.  Once you are logged in, you'll be given a form that allows you to enter your Loggly's account 'subdomain'.  Once you submit your subdomain, you'll be taken to Loggly to login to that account and grant access to the Alert Birds' app.  You'll be taken back to the Alert Birds' site when you are done authing the app to Loggly.

Alertbirds allows you to create an unlimited number of alerts, each of which are comprised of saved searches run on Loggly and endpoints that get called when searches return above or below a given threshold.  Alerts are run on a schedule you define, and the endpoints currently (only) support pushing events to PagerDuty's APIs for notification.  Alert Birds also allows you to trigger sound events in a  browser viewing the Alert Birds' website. 

Here's a short video on using the Alert Birds' application (sorry for saying um so much):

For more detailed setup info, you can reference our wiki page on Alert Birds.  If you'd like to trigger an alert for Kord on his blog, click here.

Apps for a Logging Platform

We've Open Sourced the Alert Birds project to help illustrate how easy it is to build a logging application on Loggly.  Alert Birds was writtien to run on Google's AppEngine service, but we'll be launching other apps soon that run on other frameworks and services.

Keep an eye out for more applications and app support coming out of Loggly in the near future.  We're working on apps like  a 3D globe for visualizing website traffic, web analytics, application reporting, and more!

No Comments   |   Leave a Comment   |   Tweet

IBM developerWorks podcast with Andy Glover

Posted 7 Sep, 2011 by jon@loggly.com in Log Management

Andy Glover (from App47) hosts a podcast on IBM developerWorks, and asked me to chat about some of the things we're doing here at loggly. I never realized how many ... pauses I take while I'm ... speaking. I'm going to pretend that this is because I'm ... carefully considering what I'm saying, rather than that I took a few too many knocks to the head back when I was playing Rugby.

Anyway, here's the link: http://www.ibm.com/developerworks/java/library/j-gloverpodcast3/index.html

Beware the strong gravitational pull of developerWorks. Its way too easy to lose hours and hours browsing through all of the awesome content there.

Thanks to Andy for letting me ramble on about the wonders of search :-)

No Comments   |   Leave a Comment   |   Tweet

DreamForce presentation

Posted 7 Sep, 2011 by jon@loggly.com in Code and Log Management

Last week at DreamForce, I did a talk about how magical Loggly is, and a few people have asked to get copies of the slides. So, to make it a bit easier for everyone, here they are...

http://loggly.com/assets/4e67ec60dabe9d63dc002074/dreamworks_2011.pdf

Enjoy!

I was able to demo at DreamForce, so most of the exmaple slides ("Simple search" through "uniq") never saw the light of day there. They're included here, just for the sake of completeness.

Big thanks to Heroku & SalesForce for inviting us to DreamForce to celebrate us becoming an Add-on for Heroku - we had a lot of fun, and everyone seemed to love Hoover. Then again, how could you not?

No Comments   |   Leave a Comment   |   Tweet

Announcing the Loggly Addon for Heroku

Posted 31 Aug, 2011 by Kord Campbell in Business and Log Management

I'm pleased as punch to announce the Loggly Add-on for Heroku is now in private beta!  The fine folks over at Heroku just emailed me the good news a few hours ago:

"Heroku is very excited to announce the availability of the Loggly add-on to our thousands of developers. Loggly is intuitive, easy-to-use and makes logging fun again by providing a rich set of features enabling users to search and analyze their logs."

It's been over a year since we first visited Heroku's offices to discuss providing a logging add-on for their platform users.  The result of both company's efforts over the last year is the first third-party Heroku logging add-on which leverages the power of the our highly scalable logging search engine and the sophistication of Heroku's new Logplex infrastructure.  

Simply put, it's awesomesauce in the cloud.

 

 

Solving a Big Problem

The challenge Heroku faced with customer logs centered around getting access to all of the logs out of a dyno's stack.  Heroku's stack can generate log events from the load balancer, cache, and database server, as well as logs from other add-ons, and more.  While Loggly customers have been able to send logs from the app layer for a while now using Ed Muller's super duper Logglier library, getting the remainder of the events from the stack required a rework of the way those events were routed around on the Heroku platform.

 

Earlier in the year Heroku released the first set of features based on this work, dubbing the project LogPlex and Open Sourced it over on Github.  This solution allowed Heroku and Loggly users to use the Heroku authored Logging Add-on to forward logs to a syslog port over on Loggly.  This worked well for getting access to events out of the remainder of the Heroku stack, and set the stage for Loggly to build a proper add-on for users to add to their Heroku account.

 

Without all the hard work from the fine folks at Heroku, we'd never been able to pull off writing our Add-on.  You guys rock!

 

Scaling is Hard

Scaling a sophisticated platform as a service offering like Heroku is a massive challenge.  There are brilliant people over at Heroku who have spent insane amounts of time working on scaling their platform to 100s of thousands of applications, all the while adding non-trival features like Logplex to their infrastructure.  It's a bit like changing tires on a fighter jet flying at mach 2.

 

Loggly has been spending time changing tires on jets too.  When we launched in February of this year we supported a paltry 2GB a day volumes on accounts.  In April we raised that to 4GB a day per account and in June we doubled that to 8GB a day, as well as releasing new pricing plans supporting custom volume and retention times.  Today Loggly has over 2,500 customers, and we just upped our volumes to 12GB a day per account in anticipation of our Heroku Add-on launch.  We'll continue to up our volumes over the next few months, and continue to add features which provide custom logging solutions for web applicatoin developers.

 

Be sure to keep an eye out for more Loggly features like realtime feeds and alerting real soon.

 

Keep on logging!

No Comments   |   Leave a Comment   |   Tweet

Webinar: Getting Started with Loggly's Cloud Based Logging Platform

Posted 22 Jul, 2011 by Marie Schultz in Log Management

Getting started with Loggly's cloud based logging platform has never been easier. Kord Campbell, CEO/Founder walks you through the simple set-up process. You can follow along and set up your account within minutes in this interactive webinar.  Be on the look out for more webinars coming your way soon. 

No Comments   |   Leave a Comment   |   Tweet

Siloam Springs' Logging Story

Posted 22 Jun, 2011 by Marie Schultz in Business and Log Management

Chris from Siloam Springs from Hoover Beaver on Vimeo.

Christopher Hobbs, senior system administrator for The City of Siloam Springs, Arkansas, talks to Kord about how he uses Loggly to debug and troubleshoot the dizzying array of systems he maintains for the city and police and fire departments. You can follow Chris and Siloam Springs on Twitter, and browse his public repos on Github.

No Comments   |   Leave a Comment   |   Tweet

App47's Logging Story

Posted 22 Jun, 2011 by Marie Schultz in Business and Log Management

Chris from App47 from Hoover Beaver on Vimeo.

Chris Schroeder, CEO of App47, talks to Kord about how App47 is embedding Loggly into their offerings to provide analytics and troubleshooting for mobile developers. Focusing on the enormous challenge of understanding user behavior, improving the experience and troubleshooting application crashes. You can follow App47 and Chris on Twitter.

No Comments   |   Leave a Comment   |   Tweet

On the Way to Impressive

Posted 6 Jun, 2011 by Kord Campbell in Code and Log Management

I manage our Twitter stream, and for the most part I see an overwhelming amount of positive support for the product. Sometimes though, we get the occasional brutally honest comment indicating we could be doing a better job. Once such tweet came in about a week ago from @jakajancar. Jaka tweeted the following regarding Loggly's apparent lack of features:

He has a helluva point here. A lot of the things we do for scale and speed aren't readily apparent to the casual observer. It's really only when you start sending us several million events an hour and then start searching across 100s of millions events that you realize the scope of what Loggly can do. It's taken the lion's share of our time over the last 6 months to achieve scalability to handle thousands of accounts sending in up to 8GB/day each. Even though not everyone has those types of volumes, there exist accounts that do. We had to make sure we could handle higher volumes before we did something really cool with the rest of the system.

Well, it's about #@!%'ing time do start doing something cool with all this scalability.

Putting the Sexy Time in JSON

About 2 months ago Jon and I were talking to App47 about how they use Loggly. App47 is embedding Loggly in their own app, and requested a feature where we would extract a given field, index on it, and then allow narrowing of searches on only that field. We scoped the work, codenamed the project Argonaut and began cranking on it. The result is a new Loggly input which accepts JSON formatted data and on which partial or ranged searches can be done.

Back on Twitter, I received a reply from Jaka about what exactly he'd like feature-wise in Loggly.

Turns out the stuff we've been working on is very nearly the set of features Jaka asked for. Que the evil laughter.

Using the New JSON Hotness

Using the new Loggly JSON input is pretty easy. You simply create a HTTP input that is JSON enabled, and then forward a JSON structured text blob to it. Let's start by taking a look at some sample data we generated with a custom Apache logging format:

Now let's search for an event which matches status code 200 across a bunch of these suckers:

Loggly will only return events for this search which have a field named status and a value of 200. The hotness doesn't stop there though - you can also do ranged searches:

By now you are saying to yourself, that's pretty cool, but can I graph my field shizzle with it? Hell yeah you can! You can either use graph or compare with single values, or ranged values to conduct your searches. Here we 'bucket' the status codes and use the compare graphing command to get a breakdown of response codes:

Let's not stop there. You can also do the equivalent of a grep | cut | sort | uniq -c|sort -n if you use the unique command:

We're not entirely done with all the little options in and around these features, but will be adding them in the coming weeks. If you have suggestions you'd like to throw our way, we're more than happy to pretend the subsequent improvements were attributed to you asking for them! :P

So how do you JSON all your log data?

It's pretty obvious by now we're not planning on directly providing field extraction support in the product. Loggly's approach to log management has always erred on the side of simple, and field extractions are no exception. We'll be partnering with a few other providers in the coming months to get you endpoints for extracting fields from unstructured data, and also cranking out best practices for doing it yourself.

For now, if you log from your own applications, implementing a Loggly JSON input is fairly trivial: You just send us JSON on your JSON enabled HTTP inputs.

For other use-cases, be sure to keep an eye out for a follow-up post by Jordan showing you how to configure your Apache server to serve up custom log formats and using the grok tool to convert other logging formats on the fly to JSON. Grok will even forward the events to your account from a monitored file.

Impressed yet? Just wait for real-realtime feeds and our new alerting app coming out next month!

3 Comments   |   Leave a Comment   |   Tweet

Using the Loggly JavaScript Bug

Posted 10 May, 2011 by Kord Campbell in Code and Log Management

Update: I added support for HTTPs connections in the code examples below.

JavaScript bugs are all the rage. Dropping a little bit of someone else’s JavaScript in your website allows sites like Google Analytics, LoopFuse, and MixPanel to provide various analytics around the data they collect from people browsing your site.

Loggly already supports sending in logs from your webserver via syslog or HTTP, but it’s usually a bit more involved to set up and configure than installing one of these JS bugs. We figured this was a good enough excuse to get in on the JS bug action, so we whipped up a cool solution for doing this using iframes and cross-site POSTs from your user’s browsers. We stuck the JS file on Amazon’s CloudWatch service to ensure we weren’t slowing your site down, and we made it work with our existing HTTP inputs.

Setting it Up

Setting up the Loggly JS bug is pretty straightforward. Start out by creating a Loggly HTTP input which you’ll use to send events. Next you’ll need to include a script tag for loading the loggly.js JavaScript file. Just drop the following snippet into your webpage, at the top or bottom:

Once you have that included, you’ll want to include the following code, editing it slightly to include your HTTP input’s SHA-2 key as it appears on your HTTP input detail page, and setting the default logging level:

If you are using something like jQuery that provides a load event, you can drop the window.onload=function() bit and just stick the code inside whatever JS ready block you already have.

Log Crap Out of JavaScript

Assuming no other changes to your JS code, you’ll end up getting events flowing into your HTTP input which represent accesses to your website by remote clients. We’ll capture IP address and whatever else you send us as key/value pairs that you can use to search on, including browser width and height. Here’s an example of a search we ran through Ivan’s tally command to get the top IP access to our site over the last hour:

If you want to take it a bit further, you can actually log whatever you want out of your site’s JS. Here’s an example of how we could log a user’s subdomain stored in a cookie:

Using the data from your application and the power of Loggly’s search, you should be able to whip up any DIY analytics solution your heart desires.

BTW, if you using Loggly in a unique way, let us know. We’d love to chat with you, make a video, or even hire you!

Log on fellow beavers.

1 Comment   |   Leave a Comment   |   Tweet

Bebo's Logging Story

Posted 9 May, 2011 by Kord Campbell in Business, Log Management, and Startup

Aren Sandersen, VP Operations for Bebo, came over today and had lunch with us. Afterwards, we sat down and chatted about how Bebo is changing their infrastructure, manages logs, and how they use Loggly to do debugging, alerting and operational troubleshooting with Loggly. You can view the video on your iPhone via broadband, or watch the mobile version as well.

You can follow Aren and Bebo on Twitter, or sign up for Bebo on their website.

1 Comment   |   Leave a Comment   |   Tweet

Chris Wensel's Logging Story

Posted 20 Apr, 2011 by Kord Campbell in Business and Log Management

Last week Chris Wensel swung by the Loggly office, had lunch, and sat down to do a short video with me to talk about Cascading, what his company Concurrent has been up to, and to discuss how Hadoop is good for processing a crap load of logs. Check out the video below. You can view the video on your iPhone via broadband, or watch the mobile version as well.

You can follow Chris, Cascading and Concurrent on Twitter, download Cascading, and check out his company Concurrent, Inc. Be sure to keep an eye out for the new version of Cascading which is due out soon!

No Comments   |   Leave a Comment   |   Tweet

Getting Started with Loggly Tutorial

Posted 19 Apr, 2011 by inga weizman in Code and Log Management

Kord Campbell, Chief Log Wrangler at Loggly, recently did a webinar for ModelMetrics getting started with sending logs to Loggly.  Kord steps through process of creating an account, configuring inputs and servers, testing, and searching data. It’s a great way to get familiar with the system.

Loggly Getting Started Webinar w/ ModelMetrics from Hoover Beaver on Vimeo.

1 Comment   |   Leave a Comment   |   Tweet

Alerting with Loggly and Pagerduty

Posted 25 Mar, 2011 by Kord Campbell in Code and Log Management

I recently wrote a blog post about triggering Woot lights on my desk with some simple Python code and Loggly. While hooking up Loggly to an Arduino with Woot lights can be somewhat interesting and exciting, I really didn’t present a practical solution for monitoring and alerting on events such as exceptions or errors generated by your applications.

In this example I’ll do just that with about 7 lines of Python code and a call to the mighty fine PagerDuty service.

The Setup

This solution uses the Python Hoover library which is available via the cheese shop. You’ll need to install Hoover with the following commands, which may or may not include you to installing setuptools.

Once you have Hoover installed, you’ll need to figure out what you want to alert on. For this example I’m going to use my lame blog which is hosted on code Bret Taylor wrote for Google AppEngine and Tornado. I use my async logging library for logging out of AppEngine to Loggly.

I put a method in my blog which throws an error when you hit this page. This is the result of the exception when viewed in the Loggly shell:

The Code

Now we see the error, and know what to search on to find it, we need write some simple code that will run a single bucket facet search to return a numeric count of results over a given period of time and trigger a PagerDuty alert if we find anything. In this example I constrain my search to NOW-6MINUTES to NOW-1MINUTE to ensure the events have been forwarded and indexed by Loggly. Here’s the code:

Ok, so I lied. It’s 10 lines of Python. It’s also one line in your crontab file:

*/5 * * * * /usr/bin/python blogalert.py

In practice you’d want to take the date stamp of the bucket result and use it for PD’s incident key. This would keep any overlap in searches from triggering a double or false alert.

BTW, on a related note my wife keeps calling PagerDuty ‘the girlfriend’, because she texts and calls me at all hours of the night and I have to scamper off to acknowledge her advances. My suggestion to Alex the other day was for PD to implement sexy personas that I could pick that would whisper sweet nothings in my ear at 2AM such as, “Hey baby, web head 12 is down again. Would you like to resolve?” :P

Happy alerting!

No Comments   |   Leave a Comment   |   Tweet

Using the Loggly Node.js Library

Posted 20 Mar, 2011 by Kord Campbell in Code and Log Management

Loggly uses a fair amount of Node.js and it’s currently my favorite backend framework in which to develop cool mashups with Loggly. Last month Charlie Robbins of Nodejitsu finished up a fantastic library for doing Loggly searches, facet calls, and even sending in events to Loggly with Node.js. Charlie also posted about node-loggly’s release on Nodejitsu’s blog.

I’ve been talking about getting an analytics app built on top of Loggly for a while now, and figured this would be a good opportunity to try out node-loggly for querying Loggly facet info to drive graphs and charts. I’m already using it extensively for doing logging into Loggly from all my Node.js apps. Here’s what it looks like in action.

The entire example is hosted on Github and include instructions for getting it setup and going. Notable dependencies include Richard Henry’s awesome Paperboy static file module and Charlie’s library for Loggly.

The Code

The code below pulls a single Loggly facet call for the last week, compacts it into a single bucketed result, and then returns a simple JSON string to the client for handling. The last part logs use of the app itself to Loggly.

Multiple facet searches are made for the terms defined in the index.html file.

Note that I’m not extracting the user-agent fields with these searches! I’m just doing a count of events with the term Safari, Chrome, etc. in them and then graphing the results. A better approach would be to extract a sampling of user-agents over a short time range, then search specifically for those strings.

Lastly, here are the views for the app itself during this post’s first hour of life. This was generated using a ‘graph loggly-node-chart’ in the Loggly shell:

I’ll continue to use Winston for all sorts of stuff moving forward, and will post back more examples here and on our wiki.

2 Comments   |   Leave a Comment   |   Tweet

Tracking Signups with Woot Lights

Posted 2 Mar, 2011 by Kord Campbell in Code and Log Management

Since our public launch on February 2nd our signups have been steadily increasing. Last Wednesday we signed up a record 50+ free accounts in a single 24 hour period. We use a combination of Loggly itself, Google Analytics, LoopFuse and SalesForce to track our signups, but those data points aren’t readily accessible by the entire office. I thought it would be fun and informative to implement a more real-time monitoring system for our new friends.

Enter my Woot lights and a MP3 of the stock market bell.

The code for all this stuff is on Github. You’ll need to load up your Arduino with the serial.c file, which contains a simple serial protocol that brings a pin high on receiving a ‘Y’ and low on receiving a ‘N’. More information on talking to Arduinos with Python is here.

The ground wire and and the trigger pin (port 13) are at the top, but covered up in the diagram above. You’ll need a PNP transistor, readily available from Radio Shack, or Fry’s. You can also order them on Sparkfun, where you can get an Arduino if you don’t have one already.

The server code is pretty straightforward. It just sends a Y or N down the serial line to the Arduino. You can test by starting the server:

python arduinoserver.py &

Next, test the server by going to http://localhost:9090/ You should be able to turn the lights on and off from the page.

Now lets look at the facet call we’re making on Loggly:

# facet check for signups 2 minutes ago to 1 minute ago
resp, content = h.request("http://%s.loggly.com/api/facets/date/?q='/thankyou/'&from=NOW-2MINUTE&until=NOW-1MINUTE&buckets=1" % subdomain, "GET", headers={'content-type':'text/plain'} )
foo = simplejson.loads(content)
free = foo['data'].items()[0][1]

The search we’re running looks for the landing page for signups, which in our case is called ‘thankyou’. We’re asking Loggly to give us data that came in from 2 minutes ago to 1 minute ago, and put that in a single bucket, or result. Here’s that result as returned from Loggly:

superman-2:logglywoot kord$ python signup.py 
{
    "gmt_offset": "-0800", 
    "data": {
        "2011-03-03T02:12:27.715Z": 1
    }, 
    "numFound": 1, 
    "context": {
        "rows": null, 
        "from": "NOW-2MINUTE", 
        "until": "NOW-1MINUTE", 
        "start": 0, 
        "query": "'/thankyou/'", 
        "order": "desc"
    }, 
    "gap": "+1MINUTES"
}

Be sure to check out the Loggly API documentation on our wiki. There are all sorts of interesting things you can build with them!

Woot on.

2 Comments   |   Leave a Comment   |   Tweet

Coolcam Your Logs with Geckoboard

Posted 31 Dec, 2010 by Kord Campbell in Code and Log Management

Nothing is more addictive than good dose of data visualization. Yesterday I stumbled across Geckoboard and I have to confess I’m totally hooked. I’ve managed to bang out a simple AppEngine-based framework to proxy Geckoboard data from Loggly’s APIs. The code is currently feeding a logging dashboard on Geckoboard where we’re tracking 404 errors, signups, wiki edits, and searches from tweets about Apache logs.

To build your own logging mashup you’ll need an AppEngine account, a Geckoboard account, and a Loggly account. As of this writing, both Geckoboard and Loggly are in private beta, but if you tweet “Hey @loggly, I’m a natural @geckoboard hacker!”, I’ll make certain you get a Loggly invite. Maybe the guys over at Geckoboard will do the same!

Getting Started

Once you get your accounts all sorted around, start by creating an HTTP input called ‘appengine’ in your Loggly account. You’ll use this input to log from your Python code on AppEngine to Loggly. Follow the directions on the wiki to create an HTTP input if you need help.

If you run your own webserver, you’ll also need to configure your syslog daemon to send us your web application logs. Again, follow the instructions on the wiki to setup file monitoring. We recommend using the latest version of Syslog-NG to monitor and forward log files to Loggly.

Keep in mind you could do all this from just AppEngine if you are already hosting your app there. Check out the logging from Google AppEngine section on the wiki for more info on logging from AppEngine into Loggly. The current code is asynchronous and non-blocking!

AppEngine Setup

Getting the AppEngine code running is pretty straightforward. Go grab the code from Github, and then save it into your local AppEngine Launcher. Edit the app.yaml file and change the name of the app to something unique. Log into your Google AppEgngine account and create a new application of the same name.

Now go edit the main.py file, and edit up a few lines:

• the appengineurl should be your appname + .appspot.com


• the logglyaccount should be your loggly account name + .loggly.com


• the instances of username/password should be the same as your Loggly user’s credentials


• the URL in the logging setup (at the bottom) should be a URL for one of your HTTP inputs

Once you are done editing, start up the AppEngine Launcher and go to File…Add Existing Application and add the new application to the launcher. Feel free to test the app locally first to ensure it’s working correctly. You’ll get a few links to test with, which will return XML.

Once you get all this working, head on over to Geckoboard, and create a new custom line chart widget. Set the widget type to custom and the format to XML. Enter a label for the chart, and then enter a URL with which Geckoboard will fetch the data from Loggly.

Edit the URL a bit to reflect the app name you picked on AppEngine, and the term you are using to search your logs. In our example, our app name is ‘logglygecko’ and we’re looking for ‘404’ across all inputs for the last hour and day.

http://logglygecko.appspot.com/graph?q=404

If you want to include additional search terms, you’ll need to hand encode the URL’s spaces. This example looks for 404 errors on .jpg files:

http://logglygecko.appspot.com/graph?q=404%20AND%20.jpg

The code also supports the meter widget. Just create a meter widget on your dashboard, and go through the same settings as for the chart widgets, except replace graph with meter in the URL.

The Sky’s the Limit

Geckoboard already has a ton of widgets for displaying valuable data from various services like Google Analytics, MailChimp and more. With Geckoboard’s custom widgets, and the data coming in from Loggly’s APIs, you can now quickly hack up a dashboard to display all that gold hidden away in your log files too!

1 Comment   |   Leave a Comment   |   Tweet

Logging Challenges and Logging in the Cloud - PodCast

Posted 16 Dec, 2010 by raffy@loggly.com in Business and Log Management

I was invited as a guest to the CloudChaser podcast with Matt Grant.

We talked about a number of interesting topics related to logging, cloud, and security.

Log Management Challenges

We discussed a number of log management challenges from log generation to security in the cloud. Here is a brief list of topics we talked about:

• We first touched upon some issues with log file generation. I am discussing the lack of logging guidelines and the problems that brings with it.


• How are logs analyzed? One of the problems it that it should really be the application owners that look at their logs. From a security point of view, security analysts should look at the overall picture. But they should not be the only ones looking at those logs. It’s impossible for them to understand all the logs on n intimate level.


• Yet another problem is understanding the logs. Visualization is an interesting way of addressing that issue. Especially for reporting and exploration or discovery.


• Large-scale log storage seems to be a problem. Is it? Make sure you setup use-case driven retention policies!

We touched upon a number of other topics. Here is a short list:

• It seems that users are moving more and more into the application layer to collect logs. It’s not just the infrastructure layer anymore!


• Availability, performance, etc. can be a great way of selling your log management budget instead of using security as a selling point.


• Obviously we talked about Logging as a Service and Loggly in specific. A lot of logs are in the cloud or are being moved into the cloud ;)


• Security and regulatory concerns for logging in the cloud are always a fun topic. We discuss this briefly. The upshot is that it often isn’t a show stopper!

But hey, listen yourself!

2 Comments   |   Leave a Comment   |   Tweet

Server Density

Posted 23 Nov, 2010 by Kord Campbell in Business, Log Management, and Startup

My old friend David Myton from Boxed Ice swung by the Loggly office the other day to say howdy. I sat down with him and did a quick Geek CEO video about bootstrapping, developing product, filling the sales pipe, and listening to him being wise-beyond-his-years about raising capital.

Server Density is now growing at 20% a month, enjoyes a super low churn, and just got to break-even over the weekend. While others are just dreaming about startups, or grinding away for the man, David is here living the dream.

David Myton, CEO of Boxed Ice from Hoover on Vimeo.

David shared with me that the company will be moving to the Bay Area in a few more months, once they grow a little more and get their work stuff sorted around. They’re more than welcome to squat with us when they do!

No Comments   |   Leave a Comment   |   Tweet

Do you have a logstash?

Posted 16 Nov, 2010 by Jordan Sissel in Code and Log Management

I’m a dev ops guy, and I’ve been talking about logging problems for a long while now. Talking about storing logs. Talking about parsing logs. Talking about searching logs. Talking about reacting to logs. Now I’m at Loggly, I’m talking about it more than ever.

Today I’m releasing logstash, an Open Source tool to accomplish all that and more. You can read about the release on my blog and then go download the source and get started with it.

If you want to see it in action, I’ve uploaded a demo video on YouTube. Also, Kord and I sat down today and chatted about logstash and its future. That video is below.

Welcome to Logstash from hoover on Vimeo.

We’re looking for people to help on the project, so if you are interested, drop me a line.

3 Comments   |   Leave a Comment   |   Tweet

Business Justification for Logging - LogChat Podcast

Posted 5 Nov, 2010 by raffy@loggly.com in Log Management

<img class=“size-medium wp-image-1657 alignleft” title=“podcast-logo” src=“http://www.loggly.com/wp-content/uploads/2010/11/podcast-logo-273×300.jpg” alt="" border=0 width=“120” height=“132” />Yesterday, a friend of mine, Anton Chuvakin called me up and asked me whether I could help him out with his LogChat podcast. His co-host, Andrew Hay, was sick and couldn’t host the pod cast. I let him twist my arm and a couple of hours later, we were recording Log Cast #3.

Here are some quick highlights of the pod cast. If you are interested in the business value of logging or are interested in application logging, you should listen in!

We first talked about some news in the logging market. Not super much going on. Two highlights: Anton mentioned his logging checklist and I talked about twiggy for a bit. Python developers should have a look at twiggy, a new logging library!

We then launched into the main discussion about the business justification for logging. We talked about all kinds of things. Here are the three main points:

• Security is not always the best business justification. Especially for smaller companies. Try to address logging from an availability or service angle (performance monitoring).
• Use the same logs for multiple purposes. Logs that are useful for availability often can be used for security as well.
• Try to establish “good” logging guidelines. Application developers should try to not just log unstructured strings, but try to impose some structure, such as key-value pairs and include enough information, such as the user that executed the action, the machine it happened on, etc. For a much more detailed discussion of this topic, check out our “Cloud Application Logging” paper.

Downloads: LogCast #3, Cloud Application Logging paper

2 Comments   |   Leave a Comment   |   Tweet

Big Data Gets Bigger

Posted 13 Oct, 2010 by Kord Campbell in Business, Code, and Log Management

Edited on October 14th, for 2 orders of magnitude bad math.

Big data is big news. Big data is a big problem, and big solutions for it can drive big revenues. Because big money is involved, more and more people are writing and focusing on how big of pack-rats we’ve become. There’s only one fact everyone seems to be missing: Big is relative, after all.

Big Data in the Past

Back in the 70s when I was a kid, my family’s oil business had one of these old clunky Burroughs which my mom not-so-fondly called Maribel. Whenever you wanted to invoice someone, you would load Maribel up with the customer’s account history from paper tape and then manually enter the new invoices. When the existing tape got full, you started a new one. The tapes were yellow, about an inch across and maybe 20 feet long.

We stored these tapes in envelopes, and the envelopes were in turn stored in vertical file cabinets. The hall outside my mom’s office was lined with these files cabinets and the cabients were literarily overflowing into the kitchen because there was no more room in the hall for them. If you estimated 5 bits per line, 72 lines per foot, and 20 feet of tape, that would give you roughly 1KB of storage on a single tape. Multiply that by 1000’s of these tapes and I figure we had a total of 1-2MB of data stored in about 100-200sq/ft of space.

Lots of customers, lots of tape, lots of work, and lots and lots of data. At least lots for 1976.

Your Future Arrived Yesterday

In 1996 my future had arrived. I was running a moderate sized ISP, and found myself buying a full-height 5 1/2" 8GB drive from Seagate for my news server. It cost me just over $2,000. With that one drive alone, I could have stored nearly 300 football field’s worth Maribel’s yellow tape based data.

Just last weekend at Lucene Revolution I gave some company my email address in exchange for a 8GB USB drive. I promptly tore it apart and extracted from it’s guts a sliver of a micro SD card. I could easily fit a few thousand of those cards in the space of that old clunky Seagate drive.

Earlier this year an article in Wired quoted IDC as saying, the size of the information universe in 2009 was 800 Exabytes. IDC went on to say 2020’s information universe was expected to be a staggering 35 Zettabytes; nearly 44 times as much data as there is in existence today.

For reference, one Zettabyte = one thousand Exabytes, one Exabyte = one thousand Petabytes, one Petabyte = one thousand Terrabytes, and one Terrabyte = one thousand Gigabytes. That means a Zettabyte = a million million Gigabytes!

That’s around 3 × 10^16 times as much data as we had in our office in 1976! If we decided to store it in file cabinets filled with yellow tape, our dystopian future’s 35ZB of data would take up the surface area of 546 earths. Say what?

It reminds me of something you’d see in a Douglas Adams novel, where a thousands of small, slightly cranky robots named Maribel are forced to shovel and store yellow tape rolls until they collapse into a pile of rust several millions years later.

Smell the Data Exhaust

Data exhaust can be defined as the machine events generated when a user accesses data stored on a system connected to the Internet, such as when a user access their photos on Flickr. Hadoop Karma indicates Flickr was storing 4 billion photos by the end of 2009. In aggregate, those photos are stored on thousands of servers and are being viewed by millions of users across the globe everyday.

In a simple senario where all the photos on Flickr were viewed once each by a single user, the logs would weigh in at just over 2TB! In reality, Flickr’s log volume probably exceeds a Petabyte or more a year for just the views of the lightbox pages alone. Facebook’s numbers are even scarier. In one month they’ll store 2.5 billion photos on their system. In turn, all the people viewing those photos will generate an order of magnitude more log data than Flickr even has in all the photos they’ve ever stored.

Even though we’re in private beta at the moment, we’re already seeing combined log volumes of around 3GB a day from 15 customers. A few of our customers, including About.me and Server Density are sending us near the max of what we allow on the private beta right now. We expect those volumes to go up considerably when we launch the public beta in December, where an average customer could be sending us anywhere from 1 to 5GB a day each. It won’t take long to start referring to our data in units of Petabytes stored.

While demand for storing all those logs is accelerating along with all the data being generated, the technology behind the storage and processing of data also continues to accelerate. Within a few months time, the technology we are developing at Loggly will provide companies a way to peek into these large volumes of log data – where they couldn’t before – and allow them to see exactly what their users are doing with all that big data.

Loggly’s features for search, reporting and map reducing will make dealing with these huge volumes as trivial as stuffing a yellow punch tape into an envelope, except we don’t need a robot named Maribel to do it.

And so the Universe ended.

4 Comments   |   Leave a Comment   |   Tweet

Logging as a Service (LaaS) - A podcast

Posted 8 Oct, 2010 by raffy@loggly.com in Business and Log Management

The other day, Andreas from Nemertes posted a blog on The missing piece of cloud security?. In his blog post, Andreas talks about how there is no real solution for handling logs in the cloud. Due to the fact that Loggly has been in private beta, I can’t really say that Andreas is wrong.

Instead (or in addition to) reading the rest of this blog, listen to the podcast that we recorded last Wednesday as a follow up to Andreas’ blog entry.

Loggly is the first logging as a service (LaaS) platform.

In his blog post, Andreas mentions a number of challenges associated with customers doing their own log management in the cloud: (I took the freedom to expand the list a bit)

• Ephemeral virtual machines ask for log centralization.
• Centralization of logs creates a single point of failure.
• Installation and maintenance of a logging server takes time and costs resources (and money).
• Knowledgeable (and expensive) personnel is required to configure logging solutions and maintain them.
• Static solutions (installing your own log management tool) do not fit into the cloud model of “pay as you go”.
• Building a scalable and reliable logging solution in the cloud is hard and expensive.

These reasons and a number of others are the foundation of Loggly. We are eliminating these problems for our customers.

Let me continue along Andrea’s blog post. He moves on to talk about the benefits and use-cases for a logging as a service platform:

• security information management
• regulatory compliance, incident response and post-incident forensics
• control, visibility and resilience, while preserving “chain of custody” for audit purposes

In my view, these are fantastic use-cases for a logging as a service platform, but there are so many other uses, especially in the world of Web applications. There is a big big ecosystem around application logging that benefits greatly from a logging as a service platform. And to support that ecosystem, we will keep innovating and adding new features to our platform. A step into that direction are our new HTTP inputs that allow you to send HTTP posts containing log messages.

Want to know more? Sign up for the Loggly Beta ?

6 Comments   |   Leave a Comment   |   Tweet

Node.js SSL Server Example

Posted 25 Sep, 2010 by Kord Campbell in Code and Log Management

A buddy of mine pinged me today because he saw my name on Silas Sewell’s howto post on doing HTTP/SSL with Node.js. I emailed Silas a few days ago to have him update his cert handling to include toString() on the end of each filesystem read, and he was kind enough to give a shout out to me.

The nut of the problem was that Node.js puts a carriage return or some such cruft on the end when it reads from the filesystem. It was causing me fits with cert validation and I only found the answer by digging through the Node.js IRC channel logs. Logs, heh.

I had already expounded a bit on Silas’s solution because our signing agent uses a key chain. My buddy was also asking me for an example of how to do SSL and listen on multiple ports, so I pastebin’d him up a solution. Figure it was worth posting here too!

// includes
var sys = require("sys"),
    http = require("http"),
    net = require("net"),
    url = require("url"),
    fs = require("fs"),
    crypto = require("crypto");
  
// crypto
var privatekey = fs.readFileSync('/some/path/foobar.com.key');
privatekey = privatekey.toString();
var certificate = fs.readFileSync('/some/path/foobar.com.crt');
certificate = certificate.toString();
var chain = fs.readFileSync('/some/path/intermediate.crt');
chain = chain.toString();
var credentials = crypto.createCredentials({key: privatekey, cert: certificate, ca: chain});
  
// server object
var handler = function (request, response) {
    var content = "";
    var remoteip = request.connection.remoteAddress;

    request.addListener("data", function(chunk) {
        content += chunk;
        if (content.length > 32768) {
            response.writeHead(413, {"Content-Type": "application/json"});
            response.write("{ 'response': 'error: oversized event' }\n");
            response.end();
            return;
        }
    });
    request.addListener("end", function() {
        response.writeHead(201, {"Content-Type": "application/json"});
        response.write("{ 'response': 'success', 'length': "+content.length+"  }\n");
        response.end();
        return;
    });
};
  
// ssl'd http
var sslserver = http.createServer();
sslserver.setSecure(credentials);
sslserver.addListener("request", handler);
sslserver.listen(443);
  
// regular ol' http
var httpserver = http.createServer();
httpserver.addListener("request", handler);
httpserver.listen(80);

I should note that your keys and certs need to be readable by the Node.js server’s user. Obviously.

1 Comment   |   Leave a Comment   |   Tweet

Our Solr system

Posted 9 Aug, 2010 by jon@loggly.com in Business and Log Management

I was one of three speakers at the Lucene/Solr meetup last month, co-sponsored by salesforce and Lucid Imagination. I don’t know how anyone at salesforce with a window gets any work done, considering the view – take a look at Grant’s photo to see what I mean. Thanks to Bill from salesforce for hosting, and the guys at Lucid for organizing things. You can check out the two other talks here, as well as talks from previous meetups.

UPDATE: I’ll be doing a slightly expanded version of this talk at Lucene Revolution in Boston on October 8th, incorporating some of the stuff I talk about below.

I got a few interesting questions and comments after the talk, so I thought I’d expand a bit on what was in my slides, which were perhaps a little dense.

“Log Search is highly skewed”

In the talk, I said that the most important search data is the most recent. When you have a problem, you’re far more likely to care about what happened in the last few minutes or hours or days than what happened a month ago. Thats not say that you’ll never need to search older data, just that most of the time, you won’t.

After the talk, though, it became obvious that I should also have said that our users are likely to use search in a way that is also pretty skewed when compared to “normal” search products. Basically, we expect that most people will use the system somewhat sporadically, but that when they do, its likely to be a pretty intensive session of  bug hunting. So instead of a fairly continuous search load, we get random spikes for a small subset of all the data we have in Solr. This is actually good for us, because we don’t need to keep all of the shards for all of our customers “hot” in Solr.  When a customer shows up, we can warm their data quickly, and let Solr and the filesystem cache do their thing to deal with shards that haven’t been used for a while.

The most important point here is that the overall system is going to be spending the vast majority of its resources on indexing, rather than searching. I can’t give you numbers, but if we end up spending anything more than about 5-10% of our cycles on search, I’ll be very surprised. This is not your typical consumer search product.

0MQ

I talked a bit about 0MQ, and said that we chose it primarily because its fast and lightweight, even though its possible that we could lose data if things break. I clarified this a bit in a comment on Sarah Allen’s blog because I want to make sure the message is that 0MQ is awesome, not that it loses data. Here’s the guts of what I said…

I wanted to clarify one point in your writeup, though, to make sure people don’t get the wrong idea about 0MQ. Yes, our implementation of 0MQ has a potential “leak”, where we can lose messages, but its a very uncommon case, and the impact is small. Specifically, if one of the solr nodes dies hard, we potentially lose any events that were sent to it in the last batch (0MQ batches to minimize comms overhead). In steady state, 0MQ is rock solid, 100% reliable, and faaaaaast.

 

Pieter (at iMatix) and I are currently discussing ways to solve the hard death problem, and I don’t anticipate it being a problem very long. As I said in the talk, 0MQ is unbelievably cool – if you haven’t got a project that needs it, make one up!

We sponsored some work to get the SWAP functionality in version 2 of 0MQ, and I’ve been blown away by the guys at iMatix – they really want 0MQ to work, and work well. My  throw-away comment prompted an email from Pieter asking for more details, and, as I said to Sarah above, we’re already looking at how to fix it.

Oh, and in case you’re wondering how fast a one-armed paper-hanger is, take a look at what The Word Detective says about it (scroll down till you see the “You missed a spot” section). Maybe I should have used “flat out like a lizard drinking” instead?

Sharding

The way we create shards by indexing, then merging, then merging again and again and again raised a few questions that are worth repeating…

To recap, we build small (5 minute) shards on our hot indexers. When we stop adding events to them, they get merged with older shards until we hit another size limit (30 minutes). They then get merged with even older shards, until we hit the next time limit (4 hours). And so on up the chain until they cap out at a week long. Along the way, we push indexes from box to box, to balance the load on the system as a whole.

The first question is fairly obvious: Why?

At first glance, it seems like we’re just creating work for ourselves. Surely we could just build the shards and use them as is, right? The problem is that we would have a lot of 5 minute shards floating around the system, and we already know that Solr starts getting cranky when you run a lot of cores in a single instance. So, why don’t we just build bigger shards? The issue there is that with the version of Solr we’re using, we have to reopen the index to make new data available, and we currently do that every 10 seconds (hence the “NRT + SolrCloud = Our Nirvana” in my slides). Since we have to do this, we’d end up with too many segments in the hot index, or (if we’re not careful with our merge factor) a lot of automatic merging that means that the hot index becomes unavailable for updates for too long for my liking. So, we got pushed into this approach by something that I’m hoping will soon be a thing of the past. I’m really looking forward to Michael Busch’s talk at Lucene Revolution which promises to remove the “N” from NRT. I’m not sure what is better than nirvana, but I’m hoping to find out soon

We may have been forced into doing things this way, but there is a lot of value in the model we have. In some ways, we’re taking over a part of Lucene (merging) that has been absolutely  invaluable, but can sometimes be a little difficult to control. We now have complete control over when and where indexes get merged. I probably should point out that we deliberately don’t do any merging on the 5 minute shards, and that we’re careful with the merge parameters on the larger shards to make the merges that do happen as efficient as possible.  The model also gives us a very simple index naming scheme based on time, which means we always know exactly where to find data for a time-constrained query. More on this in a bit…

The next question (from the meetup) was what is the overhead of all this merging?

Rather than give numbers, its worth thinking about whether we’re actually doing anything more than Lucene already does when you start building big indexes. I think the answer to that is that we’re actually just exposing and taking over the automatic behaviour, rather than doing something “extra”. So I think the real overhead is close to zero. Compared to building a bunch of shards in parallel using Hadoop, we’re certainly doing more work, but most of the Hadoop based systems I’ve looked at are geared more towards building indexes from a large existing corpus, rather than dealing with a real time stream.

My final comment on this is that since its all completely configurable, we’re not locked into any of the times I’ve mentioned above. Maybe when we move to NRT, or RT, we can bump the hot shard size up to hours or days, assuming that we’re still in control of merging. We shall see…

Constructive Laziness

Circling back to the first section, where I talked about how skewed we expect our search to be, the time-based shards gives us a very clean way to limit the impact of our search requests. Since we can constrain a search to a specific time period, its easy for us to identify which indexes we need to hit to satisfy the search. Our ideal search is for something in the last few minutes, which can be entirely served out of one or two of the five minute shards. We may have gigabytes or (hopefully) terabytes of index data for the same customer sitting around on our system, but if we can satisfy their request by hitting two small, heavily cached cores, then we’re in great shape. I wonder if life will be so kind to us?

Random aside: Synchronicity

Every now and then, things just come together in strange ways. A couple of weeks ago, Kord and I talked with Diego and Santiago from Flaptor, who are working on IndexTank. Diego and I were at LookSmart together years and years and years ago, but thats not the synchronicity. As we were talking, Diego said they were working on a “Nebulizer” which does automatic distribution of their index in the cloud. The day before the meeting, I’d pulled all of the code that deals with this in our system into a class named “TheDecider” (I’m still wrestling with a way to make misunderestimate() a useful method in this class). That evening I went to a NoSQL meetup, and met someone who is also working on the equivalent for their system. Maybe there is something in the air?

4 Comments   |   Leave a Comment   |   Tweet

Cross-Domain AJAX Calls To Query Loggly's APIs

Posted 27 May, 2010 by raffy@loggly.com in Business and Log Management

Last week I started playing some more with the Logging APIs from Loggly. For the first time I started embedding AJAX calls to the API into a Web application running on an external domain. Well, guess what happened? The browser barked at me telling me that I couldn’t execute a cross-domain AJAX call. I guess from a security perspective, that makes a lot of sense. However, I started thinking about how I could overcome this problem. The one way that I could have done it was not to use AJAX, but write some code server-side that would fetch the information format the Loggly API and then present it back to my Web application. I could even expose the information as an end point on the same domain that I then query from my application (see Figure).

Well, this seemed wrong. Why did we just design a really nice, RESTful API and then developers who want to use it have to build a server-side wrapper first. This didn’t make sense to me. So I kept digging. Fortunately, I found the solution. It’s called JSONP (JASON with Padding). Here is how it works and how you can leverage it in your own applications.

Let’s assume I am building an application at labs.loggly.com that will access the API located at loggly.loggly.com. With jQuery, my AJAX call looks as follows:

$.ajax({url: "http://loggly.loggly.com/api/search/?q=ntp", username="guest", password="loggly", ...})

Now, if you do this, you will get the cross-domain error. However, if you just slightly change your call to include an extra parameter, it will succeed:

$.ajax({url: "http://loggly.loggly.com/api/search/?q=ntp", 
    username='guest', password='loggly', 
    dataType:'jsonp',
    success: function(data) {
        flare = data['data'];
    },
    error: function(XMLHttpRequest, textStatus, errorThrown) {
        alert(textStatus+" - "+errorThrown);
    }
})

Note the newly added dataType parameter. That’s it? Yes, that’s it. It will work like a charm. No more cross-domain security issues. What basically happens are two things. First, the AJAX request that is executed has one more extra query parameter: &callback=?, where the question mark is some string that jQuery randomly generates. The second thing that happens is on the Loggly side. If the callback parameter is present, Loggly does not return the plain JSON element that you would expect, but it wraps it in a function call. Something like:

jsonp12312312({data:{"May-20-2010 12:13:45": 2"}, numFound: 1})

The next thing that happens is that when your browser gets the answer back like this, it will try to execute the function called jsonp12312312. jQuery internally handled that for you by creating a function hook for that function that points to the success function provided to the AJAX call.

That’s really it. We are looking forward seeing your applications that are using the Loggly APIs!

By the way, Loggly is using Django Piston for handling the APIs. The library automatically handles JSONP responses when a parameter called “callback” is present!

No Comments   |   Leave a Comment   |   Tweet

A Logging Library for Django - How We Log at Loggly

Posted 22 May, 2010 by raffy@loggly.com in Code and Log Management

In my last blog entry, I showed you how you can enable logging in Django 1.2. Now we are going to look at the logging library that we built for Loggly to simplify the task of logging in our own Django application, the Loggly Web interface.

Here is how we log from within our application:

from loggly.logging import *

error({'object':'input','action':'create'})

That’s it. The above code creates the following log entry:

Mar 18 15:34:03 app loggly: severity=ERROR,user=logdog_zrlram,request_id=
08BaswoAAQgAADVDG3IAAAAD,object=input,action=create,status=failure

The logging call expects a dict of key-value pairs. This is to enforce key-value based log entries that make it easy for consumers to understand what a specific value means. Without the inclusion of a key, a value is more or less useless. In the example above, note that I only provided two keys: object, and action. However, the log entry contains a number of other data items. Those items are automatically added to the log entries by our logging library without burdening the developer to explicitly include them.

It is probably time to show you Loggly’s logging library:

import logging
import inspect

DEFAULT_LOGGER = 'loggly_web'
logs = None

def logHelper(rest=None, request=None):

    global logs
    output = list()

    # get the logger
    if not logs:
        logs = logging.getLogger(DEFAULT_LOGGER)

    # Loop through all the stack frames until you find the request
    stack = inspect.stack()
    for frame in stack:
        if frame[0].f_locals.has_key('request'):
            request = frame[0].f_locals['request']
            if request is None:
                continue
            # there is a request object
            if hasattr(request,'user') and hasattr(request.user,'username') and len(request.user.username)>0:
                output.append("user="+str(request.user.username).strip())
            if hasattr(request,'META') and request.META.has_key('UNIQUE_ID'):
                output.append("request_id="+str(request.META['UNIQUE_ID']).strip())
            # we found the request object. Get out of here
            break

    # getting input dictionary and appending
    if rest:
        for key in rest:
            output.append("%s=%s" % (str(key.strip()), str(rest[key]).strip()))

    ret = ",".join(map(str, output))
    return ret

def info(rest=None, user=None):

    msg = logHelper(rest, request)
    logs.info(msg)

def error(rest=None, user=None):

    msg = logHelper(rest, request)
    logs.error(msg)

Note that this is only an extract. Download the entire library if you want to use it in your own code. Here are some important things the code does:

• line 17 to 29: This part of the code inspects the call stack to check whether there is an HTTP request object somewhere. The request object contains the username for the session and that is what we automatically extract . This frees the user from manually adding that information to the logging call. Automation is good!


• line 26 and 27: We are using UNIQUE_IDs in Apache. In order to track a request from the Apache logs down into our application, we include that same ID into our Django logs. This is a huge win for associating Apache logs with our application logs.


• line 32 to 24: All the dict entries are added as ‘key=value’ pairs to the log entry. So you can log any key you want.


• line 39 to 47: These are the calls that you use in your code. Note that you can add a user field, which overwrites the username from the request. In some cases that is necessary and useful.

Let us know if you are using our library. I would love to hear back from you. I will post another blog entry later, where I will be talking about how to patch Django itself to do some more logging. We will be looking at how the authentication methods can be extended.

The links:
Django 1.2 Logging Patch
Loggly Logging Library

2 Comments   |   Leave a Comment   |   Tweet

Suffering SaaSitash

Posted 15 Apr, 2010 by Kord Campbell in Business and Log Management

Dave Rosenberg posted an opinion about cloud based logging yesterday on his Software, Interrupted blog. Dave starts out by mentioning Gartner predicted IT would spend more money on private cloud than public cloud through 2012. Here’s the exact quote from Gartner:

“Despite the economies of scale offered by public cloud providers, private cloud services will prevail for the foreseeable future while public cloud offerings mature, according to Gartner, Inc. Through 2012, IT organizations will spend more money on private cloud computing investments than on offerings from public cloud providers.”

This statement is a bit like NASA doing a press release announcing the moon is continuing to orbit the earth. Wow! The moon, still here next year? That’s awesome. Of course IT is going to spend more money on virutalization for the next few years. The success of the private cloud can be attributed to the fact virtualization has been around for a good while now, and is finally being pressed into mainstream use behind the firewall. Shoot, I think I was running Wine on some of my Linux boxes back in the mid-90s, which means virtualization has been commercialized for at least 15 years at the least. The idea of virtualizing an OS goes back well into the 60s. Come to think of it, so do I.

The public cloud, specifically IaaS and SaaS, is a grouping of emerging technologies. We’re just now starting to figure out how to wield it correctly for new business models. Poking holes in it at this point is simply rabble rousing by companies who’s business models are threatened by it and people who don’t understand it or have a use for it.

It’s a Complicance

Guy Churchward tries to make some good points in his talk with Dave, but at the end of the day, LogLogic is mainly an appliance vendor, and not only do they have big-time COGS to worry about, they also have to figure out how exactly a cloud customer is going to deploy their box on Amazon’s EC2 service. (Hint: They aren’t.) While you might be able to send logs back out of the cloud to an appliance behind the firewall, it’s unlikely to make economical sense to do so in the long term.

While there is a valid point in calling out cloud concerns, security itself is ALWAYS a concern, regardless of whether you run in the cloud or in your own datacenter. Frankly, with Loggly I’m likely better at storing and securing your logs than you are by yourself in your own data center, mostly due to the fact I’m under pressure by multiple people like you to provide a service which is expected at the outset to be secure. It’s no different than the pressure that Google has on them for securing your email, SalesForce for securing your leads, or Amazon securing your credit card info. We’re all culpable here for the security of your data.

Additionally, not all that cloudy data is created equal. A lot of the companies running in the cloud today are web based app companies, and the data they generate is often times very public in nature and not at all affected by compliance concerns. Do you think some user on Flickr cares if I stole all their comments? What about getting access to all those juicy tweets of mine? Oh wait, those are already in the Library of Congress. Nevermind, false alarm!

When IT Rains IT Pours

Log file data is already one of the largest sets of data on the planet. Logging alone in the public cloud is going to be absolutely staggering over the next few years. These trends are being driven by people switching to SaaS based applications, in turn who’s infrastructure either requires the elastic capabilities only the public cloud can provide, or who’s price point can’t be matched by private cloud offerings.

The elastic nature of these infrastructures means the logs which they generate need to be collected and stored in centralized location before the box that generated them disappears. There are many types of logs which are valuable to a company for understanding their business, and not so valuable for those data-thieving ruffians everyone keeps talking about.

While the security access data or net-flow information from public cloud vendors might alleviate the concerns of some consumers, I think there are much higher value adds to these offerings by being able to power availability and analytics services around a company’s application via a log file storage platform.

While the private cloud may continue to orbit peacefully for the next few years, the use of it for web based services will decay eventually, and it’ll be regulated to the more mundane stuff like storing my dental records and tracking my orders over on RadiatorBarn.com.

BTW, I’m still waiting on my radiator, Burton.

1 Comment   |   Leave a Comment   |   Tweet

Visualizing your Data in the Cloud with Loggly and HighCharts

Posted 26 Mar, 2010 by raffy@loggly.com in Code and Log Management

A short while into writing code for the Loggly interface we decided that we needed some eye candy. Given my background in visualization, I was keen on providing our users with an experience that helps them understand their data in an intuitive way.

Over the last few years I’ve been looking into a ton of visualization libraries for the Web. In the past, if you had asked me what library to use for generating charts on your Web site, I would have said, “Use Flash”. While there are a number of interesting Flash libraries out there, the landscape has shifted significantly in the last year. Everyone is moving to JavaScript. After some research, I opted to use a JavaScript charting library called HighCharts. I tried a bunch of other canvas-based libraries, but let me tell you without hesitation, HighCharts rocks.

I am going to show you how we are using HighCharts and how I implemented zooming to dynamically reload more event data on the fly. With any charting library, if you keep zooming in on a chart, it will not progressively load more detailed data. At detailed zoom levels you end up with a small range of data in your graph. Basically if you view a day’s data first, and then zoom into a specific minute, you would only see one data point.

To start, here’s the JavaScript I use to display a chart:

var parse_date = function(data) {
    var result = [];
    $.each(data, function(key, value) {
        var re = new RegExp(/(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:\.(\d+))?/);
        var date = re.exec(key);
        if (date[7] == undefined) {date[7]=0;}
        var real_date = Date.UTC(date[1], parseInt(date[2])-1,date[3],date[4],date[5],date[6],date[7]);
        result.push([real_date, value]);                   
    });
    return result; 
}

chart = new Highcharts.Chart({
    credits: { enabled: false },
    chart: {
        renderTo: 'activity',
        defaultSeriesType: 'area',
        margin: [10, 20, 40, 55],
        zoomType: "x",
            events: {
                selection: function(event) {
                    // change the time frame to be searched
                    var start = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].min);
                    var end = Highcharts.dateFormat('%Y-%m-%dT%H:%M:%SZ', event.xAxis[0].max);
                    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/?" \
                        + "q=inputname:logglyapp&starttime="+start+"&endtime="+end \
                        + "&facets=True&buckets=24",
                        success: function(data) {
                             chart.xAxis[0].setExtremes();
                             chart.series[0].setData(parse_date(data)); 
                             // fix the reset zoom button
                             $('.highcharts-toolbar').click(resetZoom);
                        },
                        error: function(req, text, error) {
                            $("#err").html("Reload error!");
                        }
                    });
                }
        }
    },
    xAxis: { title: { text: 'Time' }, type: 'datetime' },
    yAxis: { title: { text: '# Events' }, min:0, 
        plotLines: [{ value: 0, width: 1, color: '#808080' }]
    },
    tooltip: { formatter: function() {
            return Highcharts.dateFormat('%B %e %Y %H:%M:%S', this.x) + '<br/>'+
            '<b>'+this.y+' Events</b>' }},
    plotOptions: {
        area: {
            dataParser: parse_date,
        }
    },
    series: [{ id: 1, name: 'search', 
        dataURL: 'http://subdomain.loggly.com/api/search/'
            + '?q=inputname:logglyapp&facets=True'}],
    title: { text: 'traffic last 24 hours' }
});

var reset_zoom = function() {
    // requery for the original data:
    $.ajax({ type: "GET", url: "http://subdomain.loggly.com/api/search/"
        + "?q=inputname:logglyapp&facets=True",
        success: function(data) {
           chart.toolbar.remove('zoom');
           chart.xAxis[0].setExtremes();
           chart.get(1).setData(parse_date(data)); 
        },
        error: function(req, text, error) {
            $("#err").html("Loading error!");
        }
    });
}
});

Let’s have a quick look at the code. There are two things I want to communicate here: 1. The code I used to display a HightChart graph and 2. The way I am using Loggly’s APIs to query the data.

I mentioned the special zooming that I implemented. Take a look at lines 20 to 39. This is the function that handles zooming, and it is where I am reloading the more detailed data. I set the new start and end dates (lines 23 and 24) and then I am querying the Loggly API with the new timeframe (lines 25 to 27). Upon success – this is important – I am using the chart.series⁰.setData() method to set the new data for the chart. The next line overwrites the default button or a link that lets the user zoom out again (lines 32). Note: because you are implementing your own zoom, the default “reset zoom” button from HighCharts will not work anymore and you have to implement your overwrite it with your own function to reset the chart.

The function dealing with the reset functionality is on lines 59 to 72. It does nothing else than query the Loggly API for the original data (I am passing no time parameters) and setting the data just like the previous call. The other thing you have to do is in lines 64 where you need to remove the HighCharts default “reset zoom” link and reset the extremes (line 65).

Moving on, we’ll briefly discuss the way I’m using the Loggly API

. If you’d like to use it, you need an account with us. We are currently in private beta, therefore you will need us to give you access to the beta program in order to do so. Email if you want an account to play around with! Back to the code. Make sure you replace the with your actual subdomain. Now that this is out of the way, you can query the API by simply making a GET request to: /api/search. You pass the q parameter with your query. In my example I am getting all the data from my input with the name logglyapp. To get timeline data, you’ll need to pass the parameter facets=True into the call. This will give you counts for time buckets.

To make everything work together, you need one more piece: the date_parse function. You need this part because the Loggly API returns the data with real human readable timestamps and HighCharts wants UTC encoded timestamps. The function on lines 1 to 11 takes care of converting the time for you. Just copy it.

I hope this was useful. Let us know if you are having trouble with any of this. We are looking forward hearing about your graphing endeavors.

If you look at my del.icio.us feed, you’ll find a bunch more visualization and charting links.

4 Comments   |   Leave a Comment   |   Tweet

How to use RightScale APIs with Python

Posted 17 Mar, 2010 by raffy@loggly.com in Code and Log Management

I have been quiet for long enough on this blog. It’s time for me to share some things that I learned in the last few months while I was working on Loggly’s Application layer. Lately, I spent some quality time with Django and consequentially Python.

What I want to focus on today is our integration with RightScale. At Loggly, we use RightScale to manage our AWS instances. Loggly runs three types of servers. (Well, I am simplifying). We have a proxy tier which receives your log messages. The proxy tier, which is basically a bank of machines, forwards the messages to the indexing back end that runs Solr. The third group of machines are the Web or application servers. When a new proxy box comes online, the RightScale management interface knows about the box. I had to know about thse proxies on the application tier (i.e., within Django) as well. How do you do that?

The first solution would be to have the proxies register with Django, as soon as they get online. What happens though when they go down or are taken offline? Seems complicated to keep track of that. Another solution would be to periodically poll the proxies from Django. Not very nice either.

My solution is much more elegant. RightScale has two features that helped me out. The first one is machine tags. Each proxy server is labeled as such. (See Machine Tagging). Secondly, I am using the RightScale API to figure out how many proxies I have and what their IPs are. (As a side note, the RightScale APIs are in Beta right now. There might be changes or improvements coming down the pipe.)

I struggled for quite a bit with using the RightScale APIs out of Python. Here are some things that I learned the hard way and you might find helpful:

Using the API to query all your machines in a specific deployment:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
https://my.rightscale.com/api/acct/[account]/deployments/[deployment_number]

Note how you have to add the extra header to request version 1.0 of the API.

Here is how you get all the machines that have a specific tag. Note the structure of my tag! I set role:proxy=true. You need to use this hierarchical model!

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] -d'resource_type=server' \
-d 'tags[]=role:proxy' https://my.rightscale.com/api/acct/[account]/tags/search.js

Want JSON output instead of XML, add “&format=js” at the end of your request!

Now, from the response, you would think you could just use that HREF to query an individual server. Wrong. That doesn’t work. You have to add “/settings” in order to make that work:

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
https://my.rightscale.com/api/acct/20184/instances/[instance_id]/status

Here is how you set a tag on a server: (Note: If you change the tag in the user interface for a running server, it will not take effect. Only if you start a new server of that type, will the tag be there. Unlike the API call, where you can set a tag on a running machine).

curl -H 'X-API-VERSION: 1.0' -u [user@domain.com]:[password] \
-d 'resource_href=https://my.rightscale.com/api/acct/[account]/servers/[server_id]' \
-d tags[]=role:proxy=true https://my.rightscale.com/api/acct/[account]/tags/set

The part I struggled with most was how to call the API from within Python. Turns out httplib2 expects the Web server to respond slightly different than the RightScale server is. If you are using the following code, you will not be able to connect:

h = httplib2.Http()
h.add_credentials(user,password)
response, content = h.request(url, headers=headers)

httplib2 will connect to the Web server without sending the credentials. Only if the server challenges the client to use auth, it will then send the authentication headers. And this is precisely what RightScale is not doing. Therefore, you have to do the following in order to include the authentication headers in the first request already:

h = httplib2.Http()
import base64
base64string = base64.encodestring('%s:%s' % (user, password))[:-1]
headers['Authorization'] = "Basic %s" % base64string
response, content = h.request(url, headers=headers)

Credentials are an interesting topic. I ended up creating a separate user in the RightScale interface that I am using for the APIs. Don’t be fooled though. These credentials still let that user log into the Web interface. I hope that RightScale will add a capability such that I can have a user that can only use the API.

I hope this helps you getting off the ground a bit quicker when using RightScale. Let me know how it goes. You can also find me on Twitter: @zrlram

1 Comment   |   Leave a Comment   |   Tweet

Django Middleware Munging

Posted 4 Dec, 2009 by Kord Campbell in Code and Log Management

We’ve been hitting the code pretty hard of late at Loggly, and the beta is really starting to take shape on the development servers.  There’s lots to do, of course, so we’ve taken to using Unfuddle to track tickets, host our repository for code commits.  Later on we’ll use Unfuddle’s APIs to help track customer’s feature requests and tickets.  Here’s a screen cap of our latest commit timeline:

One of the things you’ll notice when you use Unfuddle is the presence of a subdomain in the URLs you use on the site.  Our subdomain is ‘loggly’ on Unfuddle, and we  log into our project area by going to http://loggly.unfuddle.com/ (no, you can’t check our code out).  This type of customer segmentation allows for multiple unique usernames per customer, but doesn’t require a unique username site-wide.  For non-SEO sections of the site, this is a perfect solution.

We are taking a similar approach with Loggly, where a user will sign up for an account and define a unique customer identifier (we’re kicking around calling this a “mill”), which will then be mapped to a subdomain on the system.  So, for example, if Foobar, Inc. were to sign up for a Loggly account, they would access the site via http://foobar.loggly.com/, and then could create any number of user/pass combinations they wanted to access their company’s log resources.

The only problem with this approach is that we use Django, and their built in auth system (which is fantastic, BTW) doesn’t really have facilities for this type of functionality.  While we could certainly hack the Django auth system by writing our own multi-tenant auth module, it would take away from more pressing issues – like launching the beta!

Enter the Middleware Solution

One way to solve this is by munging the subdomain and username together, which provides a unique system-wide username. If, for example, you were to log in as steve under foobar.loggly.com, then we’d stick them together to be something like “foobar_steve”. Obviously we can’t have everyone remembering this long monstrosity for their username, so we’ll need to munge the subdomain off the URL and the username the user types in to get the correct combination to send off to the auth system.

Thankfully Django provides a super-easy way to add middleware to a project. By injecting a small piece of code into the request from the user’s browser, we are able to do our on-the-fly transformation before the auth system takes over. Nobody is the wiser because we can modify the display name code in the profile model to show the “normal” username to the user. Here’s what the result looks like:

settings.py:
...
MIDDLEWARE_CLASSES = (
'loggly.profile.MungeMiddle.MungeForMillMiddleware',
...
)
...

MungeMiddle.py:
class MungeForMillMiddleware:
    def process_request(self, request):
        if request.POST.has_key('username'):
            data = request.POST.copy()
            user = "%s_%s" % (request.META['HTTP_HOST'].split('.')[0], data['username'])
            data['username'] = user
            request.POST =  data

When a request comes in, we pull out the POST data and make a copy of it with .copy(). We then munge up the username with the subdomain out of HTTP_HOST, and then set the POST data to forward on to the rest of the stack. We don’t do this for all requests, just ones with the username set, so it’s lightweight enough for production use. We end up sticking the shorteded version of the username into the profile table, and use it for display.

So there you have it. A 5 minute fix for a 5 hour problem. I’m sure there are more elegant solutions to doing subdomain segmentation with Django’s out-of-the-box auth system, but frankly we don’t have time to stop and code them up. We’re bent on getting our beta out as soon as possible, and if it requires hacks like these to do it, then so be it! Release early, release often.

2 Comments   |   Leave a Comment   |   Tweet

Coolcam for Your Log Files

Posted 11 Sep, 2009 by Kord Campbell in Code and Log Management

I’ve been talking about coolcams for years as a way to help quickly show off a product’s features. Coolcams aren’t meant to be useful – they exist simply to entertain and engage your audience. It’s like an elevator pitch for your product demo.

Last year I did a coolcam mashup based around Poly9’s Flash Globe and events from my web server’s log files. I never did get around to publishing the code, but the URL was accessed 100s of times by the sales guys I worked with. If it ever was down, or broken, I’d get an email from then in minutes. As it turns out, a lot of them used the globe to start conversations with their customers.

Loggly Globe
Flash forward to present day. I’ve completely rewritten the code and put it up on Loggly’s site to share with everyone. The way the globe works is pretty simple. Using the web.py framework, it starts a web server which does two things. First, it serves up the HTML to your browser, which includes the Poly9 globe object and the jQuery library. Second, it serves up a JSON object to the page which is parsed and sent to the globe object. The code that serves up the JSON object does a few magical things for you:

• tails yor web access log file for visits


• parses out the ip address, timestamp, etc. from the log event


• takes the ip address and does a geoip lookup on it


• removes duplicate visits from a single ip address


• wraps the whole thing up in a JSON object

While Loggly Globe is hard coded to parse our logs, it should be fairly easy to use it yourself. To get started, download the tarball for Globe 1.0, and then extract it somewhere on your server:

kord@loggly> tar xvfz globe_1.0.tar.gz

You may need a couple of Python libraries installed. Assuming you have easy_install installed you can run:

kord@loggly> easy_install web.py
…
kord@loggly> easy_install httplib2

You’ll want to edit the globe.py file and modify the location of the Apache log file to point to your local log file. You’ll also want to edit the regular expression extractions to match your log file format. Here’s a line out of our logs for reference, and the corresponding extractions, most of which were pulled from Random Encounter. Make any changes you need to match the regex up with your logs.

75.101.142.96 - - [11/Sep/2009:09:19:17 -0700] "GET / HTTP/1.1" 200 6196 "-" "collectd/4.4.2" 195546

parts = [
   r’(?P\S+)‘, # host %h
   r’\S+’, # indent %l (unused)
   r’(?P\S+)‘, # user %u
   r’\[(?P.)\]‘, # time %t
   r’"(?P.)"’, # request “%r”
   r’(?P[0-9])‘, # status %>s
   r’(?P\S)‘, # size %b (careful, can be ’-’)
   r’"(?P.)"’, # referer “%{Referer}i”
   r’"(?P.)"’, # user agent “%{User-agent}i”
   r’(?P[0-9]+)’, # stuff at end
]

Now you’ll want to start the server. You can specify a port number to listen on if you want:

kord@loggly> cd globe
kord@loggly:/globe> python globe.py 8001
http://0.0.0.0:8001/

Try hitting http://yourserver:8001/json and see if you get a response back. Here’s an example of what you should see: http://www.loggly.com:8001/json. Here’s the demo again, if you just want to skip to the good stuff. Additional work could be done to integrate the code into an Lightty or Apache install to make it more permanent. You can read more about doing that on Web.py’s cookbook page.

Once we get the beta launched, you’ll be able to make mashups like these with your own log files. We’re looking forward to doing more coolcams like this with Loggly!

2 Comments   |   Leave a Comment   |   Tweet