How to search Kubernetes logs

Loggly is a great platform to send your Kubernetes logs into. While you can run an ELK setup in the Kubernetes cluster, you still have to manage it and understand how it works. If you have high volumes of logs coming in, it’s not an easy job to scale ELK for ingestion and searching needs. Using Loggly allows you to concentrate on running your applications instead of running infrastructure. Loggly will give you everything from searching and dashboarding to alerting.

Sending Kubernetes pod logs to Loggly is pretty quick. If you haven’t read my blog on that topic, you should do that first.

Once logs are coming in, you can do a quick search to pull up some logs. Type “*” into the search bar. By clicking the “+” in the Loggly interface to expand a log event, you will see logs with a similar structure below.

json:
	kubernetes:
		 labels:
			pod-template-hash: 1321724180
			k8s-app: kube-dns
		 host: ip-10-2-21-163.us-west-2.compute.internal
		 pod_name: kube-dns-1321724180-06lvt
		 container_name: kubedns
		 pod_id: 18f676ae-41ce-11e7-8f6f-0a598f34151a
		 namespace_name: kube-system
		docker:
		 container_id: fe93db65219113165e796d74e245da0c40d7fd1e7b6304557b487d3307046701
		log:
		 I0613 03:00:56.925238       1 dns.go:462] Added SRV record &{Host:jenkins.jenkins.svc.cluster.local. Port:50000 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
		 stream: stderr
		 http:
			clientHost: 35.165.209.237
			contentType: application/json

Search for all logs from a pod via a Kubernetes label

The Kubernetes Fluentd agent has helped us pull in all of the fields we want from Kubernetes and sent it over to Loggly. This makes it very easy for us to start searching through the logs for the pods that we are interested in.

Let’s say I want to get all of the logs for the “kube-dns”. I can simply put the following into the search field to pull up all kube-dns logs even if I have more than one pod running.

json.kubernetes.labels.app:kube-dns

Search for pod logs in a Kubernetes namespace

Since the fields are hierarchical, you can just walk down it to get what you want.

Let’s say we want to pull up all of the ingress logs in our default namespace. We can use this search:

json.kubernetes.namespace_name:default AND json.kubernetes.labels.app:ingress-controller

Usually you would have a namespace for, say, dev1, dev2, and/or qa1 and qa2. Each of these namespaces would have the same pod names because you are usually running the same set of pods for different levels of testing. This means that you have to filter it by the namespace and then query for the pods that you want. To do this, you use the “json.kubernetes.namespace_name” field selector to select the namespace you are interested in and then filter from there.

Using the Loggly GUI to visually search and drill down on the fields

Another entry point into digging through your Kubernetes logs is to use Loggly Dynamic Field Explorer™. Once again, we can easily search through all of our pod logs because the Fluentd collector running on your Kubernetes cluster is doing a lot of work by putting all of the labels into each log entry.

On the left-hand side, select “JSON” and then in the “Query” box, type  kubernetes. This will bring up all of the “kubernetes” field names that are available.

If you are just browsing around, you can select “kubernetes.pod_name” and see what is there:

In my demo cluster, I have a few pods like an ingress controller and Jenkins running. Clicking the pod name will give you just the logs for that pod.

Summary

Kubernetes is so popular because it has done a lot of the hard work for us. Kubernetes labels all of our pods correctly, which makes it easy for log collectors like Fluentd to aggregate it and then ship it off to a platform like Loggly to ingest. Once in there, we use the powerful GUI that Loggly has built for us to search and parse out the logs we are interested in.

Some common DevOps use cases include:

• Monitoring and analyzing ingress load balancer request rates
• Kube DNS logs and monitoring
• Setting an alert to ensure that there is always one Kube DNS pod running
• Pulling up logs on the Kubernetes master’s processes

Stay tuned for more!

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.