Building a ChatOps bot with Slack and Loggly (part 2)

In Part 1, we described the history of ChatOps and Chatbots and learned about some frameworks. We then created a simple bot that searches Loggly. We also briefly discussed Loggly Live Tail. In Part 2, we will learn how to log information to Loggly when your ChatOps bot is messaged or spoken to. We will also learn how to include Loggly Live Tail in Slack so that logs can be streamed directly from Loggly into Slack. Searching is great, but being able to see live logs in Slack is also really useful.

For code reference take a look at

https://github.com/trorabaugh/slackerlogglybot

Pay special attention to
lib/botconfig.js file
The botconfig file is used to manage all of your tokens and authentication information for both Slack and Loggly. The slackerlogglybot above uses BotKit for the underlying framework. There are a variety of other Loggly logging libraries available, but I chose winston-loggly for this project. You can learn more about Node.js logging basics in the Ultimate Guide to Logging.

Logging to Loggly when spoken to

In the TV show Lost in Space, there was a robot. The robot was codetty clunky and seemed rather dull and unsophisticated, but it always seemed to provide important information and warnings to the Robinson family. A good ChatOps bot will take commands and do what it’s told. But we want our Slack Loggly bot to remember what was said when it was spoken to. The primary reason for logging directly to a ChatOps bot is that since your development team is already in Slack, it’s efficient for them to  interact directly with the ChatOps bot to record DevOps-related tasks or what people are currently working on. ChatOps can provide code-production, production, and post-production context to the entire team at one time. You can get the bot to search for information and warnings in your system, or get the bot to stream logs that match for certain types of issues back into Slack. Who wouldn’t want a bot that tells you if your systems are in danger?

By default, the Slack bot from BotKit and the slackerlogglybot (which uses BotKit) have some very basic commands which you can read about at another time. For now, copy the slack_bot.js over to a file called <yourbotname>_bot.js. For this project, I chose slackerloggly_bot.js. Next, let’s make it so that we can add a command to the bot so that it will log whatever message we sent directly to the bot. To do this, set up the Winston library to communicate with Loggly by adding the code below to your <yourbotname>_bot.js:

(Note: If you are using the code-made slackerlogglybot above, then you just need to update the winston-loggly options in the lib/botconfig.js.)

var winston = require('winston');
require('winston-loggly');
var options = {
    "subdomain": "<your_loggly_subdomain>",
    "inputToken": "<loggly_customer_token>",
    "auth": {
        "username": "<your_loggly_user_name>",
        "password": "<your_loggly_password>"
        },
    "tags": ["<yourbotname>"]
};
winston.add(winston.transports.Loggly, options);

Once you have this set up, the Winston logging API can communicate directly to Loggly.  Awesome, but how do we get the Slackbot to log direct or channel messages from within Slack? Normally, you would use Winston to log things like code errors, events, and other information from your application. In this case we are going to take input from a user that messages the bot and submit it to Loggly. While the REST API can be used to send log messages directly, for simplicity, I chose to use the winston-loggly API because it already provides the ability to log any type of message and could also be used to later log errors in the bot itself.

Here is an example that you could add to <yourbotname>_bot.js:

controller.hears(['^note ','^log '], 'direct_message,direct_mention,mention', function(bot, message) {
    bot.api.reactions.add({
        timestamp: message.ts,
        channel: message.channel,
        name: 'robot_face',
    }, function(err, res) {
        if (err) {
            bot.botkit.log('Failed to add emoji reaction :(', err);
        }
    });
    //console.log('Logging the message', message.text);
    message.text = message.text.replace(/^log /,'')
    message.text = message.text.replace(/^note /,'')
    var dataToLog = JSON.stringify(message);
    winston.log('info', dataToLog);
    bot.reply(message,
        ':robot_face: I have logged your message.');
    });

Now go ahead and restart your bot by running the command:
~/workspace (master) $ token="<your_slack_token>" node slack_bot.js
OR if you are using the slackerlogglybot above, edit your lib/botconfig.js
, then run:
~/workspace (master) $ node slackloggly_bot.js
This will now allow us to do something like this:
@slackerloggly log <your_message>
Here is an example:

Rather than just using Winston to log code or application-level, log-related information, we can also use the same library to send direct Slack messages to Loggly.

---

Why this is useful

DevOps teams use tools like Slack or HipChat to communicate with each other about what they are doing or working on in code-production, production, and post-production. These tools are especially important for remote teams or teams that are scattered across the globe. Having a Slackbot that can send to your log management system allows you to keep track of what developers are doing and also enables an environment for all stages of a development process to communicate. We could also choose to record the output of the entire channel, messages containing specific matched words, or even use something complex like natural language processing, AI to process the messages if we chose to or something crazy like pull metadata out of cat photos, and send the log data to Loggly.

Here is a very simple and rudimentary example:

@slackerloggly log deploying to production AWESOME

@slackerloggly log Joe is leaving for a few hours

---

Streaming Loggly data to Slack using Loggly Live Tail

In Part 1 we discussed how we can search for logs in Loggly. Now we can log messages from our slackerloggly bot, but what about streaming logs directly to a channel? Live Tail provides us with the ability to stream logs directly back into Slack matching phrases, keywords, or regular excodessions. Loggly Live Tail is a command-line utility that is similar to running a tail -f on a log file that matches strings that you are looking for.

Loggly Live Tail can be found below and is part of the Enterprise version which must be enabled by your Loggly Administrator.

https://www.loggly.com/docs/live-tail/

Requirements:

• Java V7+
  • If you don’t have Java installed, never fear if you are running Ubuntu or Debian
  • sudo add-apt-repository ppa:webupd8team/java
  • sudo apt update; sudo apt install oracle-java8-installer
• Live Tail
  • Follow the commands here
    • https://www.loggly.com/docs/live-tail/

Optional Requirements:

• https://github.com/trorabaugh/slackerlogglybot
• npm install
• EDIT ./lib/botconfig.js
  • Add all authentication tokens login/passwords, etc.
  • Add Incoming WebHooks URL and Slack tokens (read below)
• npm run installlogglycli
• npm run slacklogglybot

Once you have downloaded the requirements, you will need to log into Slack and enable Incoming WebHooks. To do this go to your Slack subdomain to the Browse Apps section and search for Incoming WebHooks.

https://<your_team>.slack.com/apps/A0F7XDUAZ-incoming-webhooks

The screen will look something like the following:

After clicking Add Configuration, you will see a section that says Post to Channel. At this point select the channel or create a new channel so that Live Tail can stream logs to it.

Once you click the Add Incoming WebHooks Integration button, you will obtain a URL similar to the following:

https://hooks.slack.com/services/T1234565/B12321321/ABcdefGHIjklmnOPQRstuvWX

You will need to save this URL for using with either the slackerlogglybot ./lib/botconfig.js or the Loggly Live Tail command-line tool. To get the command-line tokens, log into your Loggly account and click the live_tail button.

https://<your_loggly>.loggly.com/live_tail/command_line

If you are trying to use the slackerlogglybot, the slackerlogglybot already handles all of the dependencies. All you need to do is run npm install, then edit the ./lib/botconfig.js and all of your Slack and Loggly tokens as illustrated below. (Note: This is a shortened version.)

BotConfigOptions.SlackOptions = {
    token: "<your_slack_token>",
    liveTailSlackURL: "https://hooks.slack.com/services/T123456/ABcdefGHIjklmOPRQstuvWX"
}

From here you just need to

npm run installlogglycli

If you are trying to build your own bot, then you will need to install the command-line client above. Once you have the Live Tail command line installed, make sure that the livetail.properties file was updated to include both your token and the Slack hooks URL. The file is located in

/tailclient-1.0.3/conf/livetail.properties

If you ran the npm run installlogglycli command, then the properties file should automatically be updated. To check take a look at the conf/livetail.properties and look for the following to either be automatically changed if you are using the slackerlogglybot, or update them manually in the conf/livetail.properties file.

tail.client.authtoken=<your_slack_token>
tail.client.im.url=<your_slack_webhooks_url>

By now you should have the ./lib/botconfig.js updated with all of your tokens and the livetail.properties file all figured out. To test out Live Tail simply run the client using the following:

./tailclient-1.0.3/bin/livetail -m “.*”

The Live Tail command has the ability to match based on regular excodessions using the -m command or ignore regular excodessions based on the -i command.

./tailclient-1.0.3/bin/livetail -m "<matcher pattern>" -i "<ignore pattern>"

Once you start the Live Tail cmd client, you should start to see logs being streamed directly into the channel where you have the WebHooks URL. Here is an example below.

Now we have hooked up the Live Tail to our slackerlogglybot, but it requires us to manually start the command-line utility. There are a variety of ways to fix this, and some that are much better and more secure than others, but for the time being, here is a small example of how you could start a stream session from the bot you are creating. Basically it creates a child process each time you send the message @slackerloggly streamlogs. Not efficient, by any means, but this is just to show how you can get it to work. So either open up the <yourbotname>_bot.js or take a look at the slackerloggly_bot.js to see how this works.

// Stream Logs to Channel Of Choice
// Using LiveTail Loggly Cmd Tool
// See the ./lib/botconfig.js liveTail
var sys = require('util');
var exec = require('child_process').exec;
function exputs(error, stdout, stderr) { console.log(error, stdout, stderr) }
controller.hears('streamlogs',['direct_message,direct_mention,mention','message_received'],function(bot,message) {
        message.text = message.text.replace(/^streamlogs /,'');
        var fullExeCmd = './tailclient-1.0.3/bin/livetail -m ".*"';
        exec(fullExeCmd, exputs);
    return bot.reply(message, 'Starting a stream on ' + message.text);
});

This will simply start the tail client and then stream all logs into the channel we selected. We could make this more complex by having the bot listen for matching regular excodessions, or ignoring matches, or the combination above or have the bot auto-generate regular excodessions based on specific key phrases, but we’ll just stream all logs matching .* as an example. Another thing to think about in the above case is how the process gets forked each time that streamlogs is called (I know, not efficient) combined with sending direct messages without proper encoding or checking for injections, etc. (makes me feel uncomfortable).

Here is an example of just starting streamlogs:

But like I said, this is just for a demonstration and what I’m doing is in another repo, so use at your own risk; feel free to fork the repo and make changes that you think are cool, or improvements in general. Here is an example that allows you to run streamlogs with matching and ignoring excodessions:

controller.hears('streamlogs',['direct_message,direct_mention,mention','message_received'],function(bot,message) {
    message.text = message.text.replace(/^streamlogs/,'');
    var matchString = "";
    var matched = message.text.match(/\s-match=.*/g);
    var matchedString = (matched && matched[0])? matched[0] : " -match=.*";
    matchedString = matchedString.replace(/\s-ignore=.*/g,'');
    matchString = matchedString.replace(/\s-match=/,'');
    var ignored = message.text.match(/\s-ignore=.*/g);
    var ignoredString = (ignored && ignored[0])? ignored[0] : "";
    ignoredString = ignoredString.replace(/\s-match=(.*)/g,'');
    var ignorString = ignoredString.replace(/\s-ignore=/,'');
    var exMatchString = "";
    exMatchString = "-m '" + matchString + "'";
    var exIgnoreString = "";
    if(ignorString != '') {
        exIgnoreString = " -i '" + ignorString + "'";
    }
    var fullExeCmd = './tailclient-' + botconfig.options.LiveTailOptions.version + '/bin/livetail ' + exMatchString + exIgnoreString;
    console.log('#######################################\n'+fullExeCmd);
    exec(fullExeCmd, exputs);
return bot.reply(message, 'Starting a stream on ' + message.text + fullExeCmd);
});

Here is the example of matching in action. Notice how I start the streamlogs before I start logging from the bot? You could start several instances to streamlogs matching different criteria. In this case I started streaming logs matching bugaru. Then I told the bot to log “this is a bugaru bugaru buggy bugaru” and then the Loggly Live Tail bot responded. This is also an example of having multiple bots all communicate and play together in the same sandbox.

---

Wrapping it up

Now we can log messages from Slack to Loggly using the SlackerLogglyBot, search for messages that were logged, and then stream logs that match certain criteria to get live streams of log data in Slack:

@<your_bot_name> log <message_to_log>
@<your_bot_name> search <message_to_log>
@<your_bot_name> streamlogs
@<your_bot_name> streamlogs -match=<msg_to_match> -ignore=<msg_ignore>

To simplify both dealing with the dependencies, and the general setup, I have created an example located at https://github.com/trorabaugh/slackerlogglybot.

How to use this project

Git clone https://github.com/trorabaugh/slackerlogglybot.git
npm install
Edit the ./lib/botconfig.js (add all of your tokens for Slack and Loggly)
npm run installlogglycli
npm run slacklogglybot

Depending on my time, level of interest, and feedback, I will be cleaning up the output for the search results, changing the search results to be more excodessive and offer more parameters, and provide different logging mechanisms.

Taking the next step with ChatOps

There are a variety of other exciting things we can do with ChatOps. We can even have bots that trigger actions to multiple bots based on the actions of others. We can build  chatbots to run specific tasks like start new production instances, take message notifications from Loggly and search for associated tags, automate tasks, submit IT tasks, handle security incidents, start or stop servers based on log-related information, gather statistics on your servers, and log information.

You could even build a bot that processes images with data to trigger other bots to do things for you like tell you when you are in danger—Danger, Will Robinson, danger!

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.