6 key criteria for finding the right cloud-based log management solution

According to Gartner, Inc, by 2020, a corporate “no-cloud” policy will be as rare as a “no-Internet” policy is today. The fast adoption of cloud-based services can be explained in different ways, such as low cost of acquisition and ownership, ease of use, painless upgrades, and economies of scale. As part of this trend, more and more enterprises of all sizes have warmed up to cloud-based log management solutions.

However, selecting a cloud-based log management solution can be a daunting task. Most vendors offer you a free trial for anywhere from 14 to 30 days. In this limited period, you must not only evaluate which solution is the best fit for your current business needs but also consider which solution demonstrates sufficient flexibility to satisfy your future growth needs.

With so much happening around your Ops and Engineering teams, it is almost impossible for you or them to dedicate the time and effort required to make a thorough evaluation. Here are six key criteria that should drive your analysis of cloud-based log management solutions.

Security

A study of 974 attendees at RSA 2017 by AlienVault found that 39% of respondents use more than 10 different cloud services within their organizations. Despite such a rapid proliferation, security still remains one of the top concerns about these services. The Cloud Security Spotlight Report 2016, a comprehensive survey of 2,200 professionals, found cloud security concerns not only top the list of perceived barriers to cloud adoption but also are increasing.

Loggly sets a high bar to keep your data safe. Our service offers secure data transmission, secure account access, and role-based access control. All your log data is automatically and permanently deleted at the end of your retention period. For further details on our security practices, see Log data and security.

Scale

As your business continues to grow, you must think of how much daily log volume will be processed and how long the data must be retained for troubleshooting or compliance. Furthermore, logs are unpredictable. Time and time again, we have seen how quickly log volumes can grow whenever some production issue arises. In fact, it is not uncommon for us to see a customer’s log volume burst three to four times during production fires or peak business activity. So it is very important that your cloud-based log management solution remains capable of handling any volume of data, from gigabytes to terabytes. Some of Loggly’s customers regularly exceed the terabyte threshold every day. Not just that, Loggly offers customers in the Pro and Enterprise plans Peak Overage Protection (POP), ensuring that data gets captured even if the customer exceeds its subscription level.

Speed

When your developers and Ops team members are troubleshooting critical production issues, slow response times can be extremely frustrating. So it is essential that your selected vendor makes your log data ready for analysis and monitoring in near real time. Loggly places a premium on ensuring that customer data is not lost and that it continues ingesting log data for thousands of customers even when they see sudden volume bursts. Not only that, you can also monitor all your log data that is being sent to Loggly in a consolidated form and in near real time using Live Tail.

Sources

As technologies evolve, API usage goes up, container services become more popular, and the number of partner integrations increases, your systems will continue to grow more complex. According to Forrester, 2017 will be no different with the further rise of microservices and the Internet of Things (IoT).

Anyone with more than six or seven microservices in production on more than 20 servers simply must centralize the log data.

Mohit Khanna Senior Cloud Architect, Datami

Read case study

Modern applications have so many moving parts that it is difficult to diagnose and troubleshoot any incident without having complete visibility through logs across all the layers of the technology stack—from network to applications. You can make it easier for your customer support and engineering teams to troubleshoot by sending all your logs to your central log management repository. Likewise, your cloud-based log management vendor must allow you to send data from a wide variety of sources. With Loggly, you can easily find scripts and libraries to send log data for more than 50 popular systems, platforms, and programming languages.

Search

The heart of any analysis and troubleshooting process is search. In the world of voluminous logs, searching for a particular event can be like finding a needle in a haystack. Your team must have the ability to search easily and quickly against both structured and unstructured data. So you must pay special attention to the search capabilities of your cloud-based log management solution. Consider how fast the searches are, how flexible the query language is, and what would be the learning curve on sophisticated searches for new employees and less tech-savvy members of your organization, such as customer support and QA. One Loggly customer, Recruiterbox, has even been able to deploy Loggly to non-technical Customer Happiness team members who provide first-line support to HR professionals.

Loggly search gives you a wide range of capabilities and makes it easy for both beginners and power users to look for log events of interest. We recently introduced GamutTM Search, which offers near-instant responsiveness when searching against massive volumes of data for long time periods. Check out this video explaining how our new search helps you search faster and reduce your mean time to resolution (MTTR).

At Stormpath, non-operations people and even non-engineers need to interact with logs. Loggly makes it easier for these people to formulate queries in the way they naturally think about them than it is with log management systems that are focused on the workflows of sysadmins and operations people.

Les Hazlewood Co-Founder/Chief Technology Officer

Savings

Finally, every buyer and organization must consider which cloud-based log management solution meets the budget and offers maximum cost savings. Cloud-based solutions can lower your total cost of ownership (TCO) as compared to building and maintaining your own solution. But how do you compare between different cloud-based vendors?

Generally speaking, your monthly bill based on log data volume and retention is the most obvious cost with a cloud-based log management vendor. But there are other activities that differ across vendors, with less obvious costs.

These activities encompass planning, building, and managing the following activities:

1. Security and access controls
2. High performance (including managing indexes, shards, and clusters)
3. Integrations with popular DevOps tools like PagerDuty and JIRA

In addition, people often overlook the need for excess capacity to handle seasonal variations, as well as redundancy for making your logs highly available. Think about the fact that the time when infrastructure is down is often when you have the most need for your logs!

Finally, you should consider the fact that open source solutions don’t take into account the ongoing costs of setting up and managing aspects of the deployment such as index management, shard allocation, configuration management, and access control.

If your cloud-based log management vendor dumps some or all these responsibilities on your Ops team, then you must account for personnel and inventory costs for all of these items. Don’t forget, there is a huge opportunity cost as well, since your Ops team is doing these log management tasks instead of focusing on your core business.

During the Amazon S3 outage at the end of February that took down several websites, all Loggly logs remained available. So think of what it would cost you to keep your logs always available and prepared for any similar situation. With Loggly, you can rest assured that all these things are taken care of so that you can focus on managing your business.

Loggly has been very well received by the whole team, and it has lowered our operational overhead. The invoice amounts are close, but that doesn’t take into account the time my team would have to spend running our own log management solution.

Jeremy Koerber Senior Systems Engineer, Creative Market

Read case study

Make the most of your trial period

Even a 14-day free trial passes by quickly. When you start your trial, all of the new features and functionality can seem overwhelming. And too often, evaluators fail to test drive all the features of a vendor under consideration during the trial period. Using our six criteria can keep you a bit more focused on finding the right log management solution for your enterprise.

Here are some tips to make the most of your free trial.

1. Send a variety of log sources into your trial account, not just a single set of system or application logs. This gives you a better opportunity to understand how the system works and to see how it will scale, not only in handling your growing log volumes but also in terms of time commitments and skills required of your team.
2. If possible, send data from production systems, not just dev or QA. Production systems will give you a more realistic assessment of the product’s speed.
3. Pick a specific microservice or function of your application to analyze. This allows you to push the product’s search and analysis tools a bit harder than if you were simply exploring or doing general troubleshooting.
4. Give access to a diverse group of team members, including customer support representatives. Unless you want your full-time job to revolve around log analysis, you need to be sure that everyone can get productive with the service.
5. Understand your real costs at scale. At Loggly, our account managers are excellent at helping customers model their costs at different log volumes and account for normal or seasonal variations.
6. Prepare your checklist of security requirements and validate them with the vendor.

Finding the right cloud-based log management solution is not easy, but with this checklist you are almost there!

You can start a free trial in less than a minute.

Interested in learning more? Check out our Log Management Buyer’s Guide.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.