Monitoring & logging for containers with Sematext
8 min read
In this blog post we discuss how Sematext integrates with IBM Cloud Container Service to provide monitoring and logging visibility of your containerized applications, as they run in production.
In the sections below, we demonstrate how to set up a Kubernetes cluster in IBM Cloud and how to set up Sematext in this cluster. IBM Cloud has monitoring and logging capabilities in the platform, but we know our customers operate in a multi-cloud or hybrid cloud environment and we are very excited to partner with Sematext, enabling operational consistency across those environments. We worked with Alen Komljen, an Automation Engineer from Sematext, to create the following content and perform the technology validation.
To see the process end to end, watch the live demo.
About IBM Cloud
IBM announced its Cloud platform in June 2014, providing users with a variety of compute choices as well as over 140 IBM and third party services. IBM Cloud Container Service combines Docker and Kubernetes to deliver powerful tools, an intuitive user experience, and built-in security and isolation to enable rapid delivery of applications all while leveraging Cloud Services including cognitive capabilities from Watson.
Every modern organization depends on software, and where there is software there are metrics and logs and, one would hope, monitoring of both of these critically important types of data. Organizations that use multiple tools from multiple vendors ultimately pay a high price, both in licensing fees and in ongoing operational inefficiencies. Sematext offers the platform that eliminates the chasm between performance monitoring and logs. Sematext Cloud (SaaS) and Sematext Enterprise (on premises) provide Infrastructure Monitoring, Application Performance Monitoring (APM), and Log Management in a single, unified solution. DevOps teams using Sematext work more efficiently, troubleshooting faster than with traditional log management and monitoring software. Sematext Docker Agent is a modern, open-source, container-native monitoring and log collection agent for Docker. It runs as a tiny container on every Docker host and provides automatic collection and processing of Docker Metrics, Events and Logs for all cluster nodes and all auto-discovered containers.
In this blog post we’ll walk you through the whole process: from creating a Kubernetes cluster with IBM Bluemix to adding Sematext centralized log management and monitoring solution. Some of the benefits of such a setup are:
A single agent for metrics, container events and logs
Automatic collection and processing of Docker metrics, events and logs
Easy correlation between metrics and logs for faster troubleshooting
No need to deploy and maintain your own ELK stack
Built-in log parsing
Log enrichment (like GeoIP for web server logs)
Log analytics with visualisation and alerts
Continue reading for a complete step-by-step guide to use Sematext with IBM Bluemix Container Service.
STEP 1 Setting up a Kubernetes cluster in IBM Cloud
After you’ve successfully created an account and logged into it, the left-hand navigation will take you to containers.
Select the Kubernetes Cluster icon. Depending on your account type, you may only have access to a lite (free!) cluster. A paid cluster will enable additional configuration parameters, but is not necessary. Detailed documentation for cluster creation is available. Set the following parameters for a paid cluster:
Machine type – a flavor with pre-defined resources per worker node in your cluster
Number of workers – 1 to N based on capacity requirements; can be scaled up or down after the cluster is running
Private and Public VLAN – choose networks for worker nodes (Bluemix will automatically create that for you if you don’t have any yet)
Hardware – clusters and worker nodes are always single-tenant and isolated to you, but you can choose the level of isolation to meet your needs (shared workers have multi-tenant hypervisor and hardware whereas dedicated worker nodes are single-tenant down to the hardware level)
Once you are satisfied with your selections, click on the ‘Create Cluster’ button.
Prefer the command line instead? The same configuration options can be integrated with your existing CI/CD pipeline. There’s detailed documentation and sample command to create cluster:
Wait a few minutes for a cluster to be deployed, go to ’Access’ page on the left-hand navigation menu and click on download link to get your kube config file. From here you can use kubectl on your localhost to manage the kubernetes cluster:
STEP 2 Deploying the Sematext Docker Agent on IBM Cloud Container Service
Think of Sematext like an all in one solution for monitoring, logging, and troubleshooting your services running on Kubernetes with only one agent. Sematext Docker Agent facilitates automatic collection and processing of Docker metrics, events, and logs for all cluster nodes and all auto-discovered containers.
You’ll want to create a Sematext account and, to have metrics and events for your newly created Kubernetes cluster, create a Docker App in Sematext. When you do that you will see instructions for launching Sematext Docker Agent with Kubernetes. You will also see a commented out reference to LOGSENE_TOKEN:
As you can tell from the inline comment, we’ll use this to collect logs. You can copy the whole presented Kubernetes config in your favorite text editor, then create a Logs App in Sematext, and replace/uncomment that LOGSENE_TOKEN with the token for the Logs App you just created. After that, a simple kubectl create command will deploy Sematext Docker Agent to all nodes in the cluster:
Wait a few minutes and you should see your metrics and logs in your Sematext Apps.
STEP 3 Keep Metrics and Logs Together
In your Logs App, you will see logs from all containers that are running in your Kubernetes cluster and in the Monitoring App you should see all the metrics. Events are automatically added with Monitoring App and you can see them on the Alerts & Events screen. If you want to customize your view, you can do it easily by creating a custom Dashboard:
Container metrics are easy to understand. You will get all of the major metrics about Docker containers in one place. CPU, memory, networking, and many others that you can filter by host, tags, image, container name, etc.
By collecting Events Sematext provides you with insight into what is happening with your containers during deployments or their scheduling to different nodes. Docker Events also carry information critical for application security, such as:
Version changes of application images
Changes of storage volumes or network settings
Deletion of storage volumes, which might cause data loss
However, digging through different kinds of logs is something that every DevOps or Developer faces during troubleshooting.
Log metadata – tags
Sematext Docker Agent has built-in and customizable log parser that turns logs into structured JSON. This makes it possible to slice and dice logs and thus gain a lot more insight about how your containers, servers, and applications are operating.
One of the great Sematext Docker Agent features is automatic log format detection. Out of the box it can parse logs from official images like:
Nginx, Apache, Redis, MongoDB, MySQL
Elasticsearch, Solr, Kafka, Zookeeper
Hadoop, HBase, Cassandra
Any JSON output with special support for Logstash or Bunyan format
Plain text messages with or without timestamps in various formats
Various Linux and Mac OSX system logs
In addition, you can define your own custom patterns for any log format you need to parse and structure. More on how to configure parser is available in Sematext Docker Agent docs. To use custom patterns in your Kubernetes cluster you need to define patterns.yml in ConfigMap and mount it in Sematext Docker Agent, as shown in these sample files.
The component for detecting and parsing log messages – Logagent – is open source and contributions for even more log formats are welcome.
STEP 4 Correlate Metrics and Logs for Faster Troubleshooting
You already saw how it all fits together and how a single agent can collect all key operational data. While it is indeed very handy to have a single lightweight agent collect all the data and to have metrics and logs right there in front of you in the same UI, wouldn’t it be cool if you could easily correlate metrics and log messages? When troubleshooting you will probably want to see if some event or log message is somehow correlated to the metric that triggered an alert notification. Sematext makes this very easy. For example, big spikes in CPU consumption could be related to requests for non-existing pages on your Nginx server. Your server could be under attack. With Sematext, you will find about those anomalies fast and block that kind of requests in the future.
If you navigate to the Monitoring App in Sematext, you can correlate metrics with logs, as well as events, all in one place:
Correlate metrics with logs and events
With a click on a button, followed by a search against logs we were able to spot a correlation between the increase in CPU usage and log messages that started appearing exactly when the CPU usage went up. Curious, isn’t it?
Compare doing this to having to use one tool or service for your metrics and another one for your logs. Nobody is good at context switching. Not even computers, let alone us humans. Sematext eliminates this expensive context switching, so you can troubleshoot faster and easier.
With IBM Cloud you can easily create Kubernetes cluster with a click of a button. You can manage your Kubernetes resources from your local machine with IBM Cloud CLI or official Kubernetes CLI (kubectl). However, adding and managing your applications is just one part of the story and this is where Sematext comes into play. The same way you deploy your apps you could add Sematext all-in-one agent which will provide you with all container metrics, events, and logs and a single pane of glass to access them all. Both services provide 30 days free trials. So if you are already using IBM Cloud Container Service and want to give Sematext a try, sign up for a free Sematext trial anytime.
Watch a live demo to see the whole process end to end.