Kubernetes Log Forwarding with Syslog

Logs help you troubleshoot issues with your clusters and apps. Sometimes, you might want to send logs somewhere for processing or long-term storage. On a Kubernetes cluster in the IBM Cloud Container Service, you can enable log forwarding for your cluster and choose where your logs are forwarded.

Using the Container Service CLI, you can forward your container logs to a syslog server with one command:

bx cs logging-config-create mycluster \
    --hostname mysyslog.example.com \
    --type syslog \
    --namespace default

The above command creates a logging configuration to send all container standard output and error logs from the default Kubernetes namespace. These logs are sent using the syslog protocol to mysyslog.example.com.

Try it out

In this tutorial, you will forward your logs to an rsyslog instance running in the same cluster.

Create a Kubernetes cluster on the IBM Cloud Container Service and wait for it to become ready. Next, connect kubectl  commands to your cluster with the following command:

eval `bx cs cluster-config mycluster --export`

Next, create an rsyslog service we can forward logs to.

Start by creating deploy-rsyslog.yaml with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: rsyslog-service
spec:
  selector:
    app: rsyslog
  ports:
  - name: tcp-syslog
    port: 514
    targetPort: 514
    protocol: TCP
  - name: udp-syslog
    port: 514
    targetPort: 514
    protocol: UDP
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: rsyslog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rsyslog
  template:
    metadata:
      name: rsyslog
      labels:
        app: rsyslog
    spec:
      containers:
      - name: rsyslog
        image: voxxit/rsyslog:latest
        imagePullPolicy: "Always"
        ports:
        - name: incoming-logs
          containerPort: 514

Then run the following:

kubectl create -f deploy-rsyslog.yaml

Then configure your logs to go to the rsyslog service.

bx cs logging-config-create mycluster \
    --hostname rsyslog-service.default \
    --type syslog \
    --namespace default

Finally, deploy a container to your cluster that generates logs. I like using a noisy pod to verify that log forwarding is working. Make a deploy-noisy.yaml file with the following contents:

apiVersion: v1
kind: Pod
metadata:
  name: noisy
spec:
  containers:
  - name: noisy
    image: ubuntu:16.04
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 10; echo 'Hello world!'; done"]
    imagePullPolicy: "Always"

Finally, create the noisy pod.

kubectl create -f deploy-noisy.yaml

Now take a look inside the rsyslog instance to see the logs.

export rsyslog_pod_name=`kubectl get pods -l app=rsyslog -o jsonpath='{range .items[*]}{.metadata.name}'`
kubectl exec -it "$rsyslog_pod_name" -- tail -f /var/log/messages

If you see some Hello world ! lines, then you have successfully forwarded logs to your rsyslog service.

To learn more, continue reading about log forwarding or IBM’s Kubernetes offering.