Kubernetes Log Forwarding with Syslog

Logs help you troubleshoot issues with your clusters and apps. Sometimes, you might want to send logs somewhere for processing or long-term storage. On a Kubernetes cluster in the IBM Cloud Container Service, you can enable log forwarding for your cluster and choose where your logs are forwarded.

Using the Container Service CLI, you can forward your container logs to a syslog server with one command:

bx cs logging-config-create mycluster \
    --hostname mysyslog.example.com \
    --type syslog \
    --namespace default
Scroll to view full table

The above command creates a logging configuration to send all container standard output and error logs from the default Kubernetes namespace. These logs are sent using the syslog protocol to mysyslog.example.com.

Try it out

In this tutorial, you will forward your logs to an rsyslog instance running in the same cluster.

Create a Kubernetes cluster on the IBM Cloud Container Service and wait for it to become ready. Next, connect kubectl commands to your cluster with the following command:

eval `bx cs cluster-config mycluster --export`
Scroll to view full table

Next, create an rsyslog service we can forward logs to.

Start by creating deploy-rsyslog.yaml with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: rsyslog-service
spec:
  selector:
    app: rsyslog
  ports:
  - name: tcp-syslog
    port: 514
    targetPort: 514
    protocol: TCP
  - name: udp-syslog
    port: 514
    targetPort: 514
    protocol: UDP
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: rsyslog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rsyslog
  template:
    metadata:
      name: rsyslog
      labels:
        app: rsyslog
    spec:
      containers:
      - name: rsyslog
        image: voxxit/rsyslog:latest
        imagePullPolicy: "Always"
        ports:
        - name: incoming-logs
          containerPort: 514
Scroll to view full table

Then run the following:

kubectl create -f deploy-rsyslog.yaml
Scroll to view full table

Then configure your logs to go to the rsyslog service.

bx cs logging-config-create mycluster \
    --hostname rsyslog-service.default \
    --type syslog \
    --namespace default
Scroll to view full table

Finally, deploy a container to your cluster that generates logs. I like using a noisy pod to verify that log forwarding is working. Make a deploy-noisy.yaml file with the following contents:

apiVersion: v1
kind: Pod
metadata:
  name: noisy
spec:
  containers:
  - name: noisy
    image: ubuntu:16.04
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 10; echo 'Hello world!'; done"]
    imagePullPolicy: "Always"
Scroll to view full table

Finally, create the noisy pod.

kubectl create -f deploy-noisy.yaml
Scroll to view full table

Now take a look inside the rsyslog instance to see the logs.

export rsyslog_pod_name=`kubectl get pods -l app=rsyslog -o jsonpath='{range .items[*]}{.metadata.name}'`
kubectl exec -it "$rsyslog_pod_name" -- tail -f /var/log/messages
Scroll to view full table

If you see some Hello world! lines, then you have successfully forwarded logs to your rsyslog service.

To learn more, continue reading about log forwarding or IBM’s Kubernetes offering.

More from Cloud

Modernizing child support enforcement with IBM and AWS

7 min read - With 68% of child support enforcement (CSE) systems aging, most state agencies are currently modernizing them or preparing to modernize. More than 20% of families and children are supported by these systems, and with the current constituents of these systems becoming more consumer technology-centric, the use of antiquated technology systems is archaic and unsustainable. At this point, families expect state agencies to have a modern, efficient child support system. The following are some factors driving these states to pursue modernization:…

7 min read

IBM Cloud Databases for Elasticsearch End of Life and pricing changes

2 min read - As part of our partnership with Elastic, IBM is announcing the release of a new version of IBM Cloud Databases for Elasticsearch. We are excited to bring you an enhanced offering of our enterprise-ready, fully managed Elasticsearch. Our partnership with Elastic means that we will be able to offer more, richer functionality and world-class levels of support. The release of version 7.17 of our managed database service will include support for additional functionality, including things like Role Based Access Control…

2 min read

Connected products at the edge

6 min read - There are many overlapping business usage scenarios involving both the disciplines of the Internet of Things (IoT) and edge computing. But there is one very practical and promising use case that has been commonly deployed without many people thinking about it: connected products. This use case involves devices and equipment embedded with sensors, software and connectivity that exchange data with other products, operators or environments in real-time. In this blog post, we will look at the frequently overlooked phenomenon of…

6 min read

SRG Technology drives global software services with IBM Cloud VPC under the hood

4 min read - Headquartered in Ft. Lauderdale, Florida, SRG Technology LLC. (SRGT) is a software development company supporting the education, healthcare and travel industries. Their team creates data systems that deliver the right data in real time to customers around the globe. Whether those customers are medical offices and hospitals, schools or school districts, government agencies, or individual small businesses, SRGT addresses a wide spectrum of software services and technology needs with round-the-clock innovative thinking and fresh approaches to modern data problems. The…

4 min read