Customized Parsing New Features: Derived Fields and Derived Tags

With Loggly, the most common types of logs are automatically parsed and broken down into fields and values. This works out-of-the-box with log files from e.g. Apache, Nginx, data that is formatted in JSON, and many other log types. But what if you have log data in a custom format unknown to Loggly? What if you want to further parse entries in one of the known formats, for example to extract a host name or a session ID from a field that also contains other information?

This is where Derived Fields come into the game. They now allow you to define custom rules through which Loggly will parse your data and break it into fields. These will then automatically be cataloged through Loggly Dynamic Field Explorer™ for easy, one-click summaries and analysis, and also allow for easier information extraction, better dashboards, and more efficient alerts.

There are three main rule types: Key-Value (defines characters as separators and delimiters, like “=”), Anchor (defines strings that precede and follow the field), and RegEx (for anything that can be matched by a regular expression).

A new fourth rule, Insert Tag, will allow you to do just that—insert custom tags based on defined conditions, so you can enrich your logs with useful information, for example to mark error messages of different types and formats as “ERR”. Needless to say, you then can make use of this new tag for all your analysis and also define alerts based on it.

Inserted tags and all Derived Fields are added as extra metadata to your original logs, and will not modify the original data. The rules will be applied to all your incoming log data after Loggly ingests it.

Rules can be named and tested before final activation.

Try It for Yourself

If you’re in a free trial now or have a Pro or Enterprise account, you can use Derived Fields right away. See the documentation for how to get started and for all the details. Users on a Lite or Standard plan will need to upgrade to Pro to take advantage of these new features. Pro plans start at $99 per month (with an annual commitment).

Not using Loggly yet? Sign up for a 14-day free trial and check out both of these new features. You get full access to the Loggly Pro plan feature set with all Loggly free trials.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.