In my previous article, Sending MQ error logs to the Bluemix Logmet service, I demonstrated how it was possible to send MQ error logs to the Bluemix logging service - Logmet - to be searched, queried and visualised.
In this final article on MQ logging and metrics, I will explain how you can send MQ usage metrics to the Bluemix Logmet service and visualise them with the Grafana web front end. We'll be using the code from Mark Taylor's article to extract MQ V9 statistics from the queue manager using his MQ Go client, and combining it with collectd, an application for collecting system statistics. This will will forward the metrics on to Logmet. collectd has various input and output plugins. We will configure it with Mark's MQ Go client as an input, and Logmet's mulit-tenant lumberjack client as an output.
The first thing we need is to install collectd and the Logmet lumberjack output plugin. Download and install the latest version of collectd for your platform from https://collectd.org/download.shtml
Once installed, download and configure the Logmet lumberjack plugin. Bluemix documents the process for doing this on Ubuntu.
Note: Before following the Bluemix article above you will need to configure your repository manager with the Bluemix repository using the command
The Bluemix article describes how to configure the collectd lumberjack plugin collectd-write-mtlumberjack with your Bluemix access token and logging token. This is much the same process that we used when configuring Logstash, issuing a cURL command to retrieve your access_token and logging_token:
curl -XPOST -d 'use
email@example.comfirstname.lastname@example.org&space=dev ' https://logmet.ng.bluemix.net/login
The returned tokens need to be added to a collectd configuration file in /etc/collectd/collectd.conf.d. When you installed collectd-write-mtlumberjack it will have installed a sample configuration file in /etc/collectd/collectd.conf.d called mt-metrics-writer.conf.sample. Rename this mt-metrics-writer.conf and edit it to have your credentials. Remember to uncomment the necessary lines. The configuration file should look similar to this:
LoadPlugin write_metric_mtlumberjack <Plugin write_metric_mtlumberjack> <Logstash> Host "metrics.opvis.bluemix.net" Port "9095" GraphitePrefix "<Space ID from cURL response>.<Optional Group Name>" SpaceId "<Space ID from cURL response>" LoggingToken "<Logging token from cURL response>" </Logstash> </Plugin>
The next step is to compile Mark's mq_coll Go application, following similar steps to those in his article. Instead of compiling the Prometheus client in his example, compile the mq_coll application:
Now we can configure collectd to use the mq_coll application as an input. Add a file called mq.conf to /etc/collectd/collectd.conf.d/, containing:
LoadPlugin exec <Plugin exec> Exec "mqm:mqm" "/usr/local/bin/mq_coll.sh" "QM1" </Plugin> TypesDB "/usr/local/bin/mqgo/mqtypes.db"
This tells collectd to invoke mq_coll.sh with QM1 as the only input parameter. You can set QM1 to the name of any version 9 queue manager you wish to monitor.
The final step is to tell collectd about the data types mq_coll will produce. This is done by adding a file listing the different types, into a .db file. If you cloned the MQ Go Git repository earlier you will find a file called <repo-root>/cmd/mq_coll/mqtypes.db. Put a copy of this file in /usr/local/bin/mqgo/mqtypes.db.
Restart collectd (sudo service collectd restart) to pick up the new configuration files. You can check that mq_coll is running using ps. By default collectd does not write logs to a file, but you can configure it to do so by adding the LogFile plugin to a .conf file in /etc/collectd/collectd.conf.d/
Once collectd is running it will automatically start pushing MQ metrics to Logmet via the lumberjack plugin. To see the metrics we will create a Grafana dashboard.
Before we do that we need a quick introduction to Graphite and Grafana. Graphite is an open source technology for storing metrics and statistics. It performs a similar job for metrics and stats that Elasticsearch does for log records. Grafana is a web front end for Graphite which allows you to query and visualise the data stored in Graphite.
Navigate to Logmet and choose Grafana rather than Kibana. You will be presented with the default metrics dashboard:
There is a useful overview of Grafana in this Bluemix article. We will recap some of the key features here.
Grafana divides the screen into rows - you can see 3 defined on the default dashboard - and then lets you put one or more panels onto each row. You can add rows, remove rows, and change their heights. There are 3 different types of panel that can be added to a row: graph; singlestat; text.
As you would expect, a graph panel lets you plot one or more data series on a graph that you can customise and format however you wish. The default dashboard has a sample graph on the bottom row, with a random number generator as the data source.
This is used to display the value of a particular metric. It is useful for displaying meter-like values such as the current depth of a queue, or the current rate at which messages are being put. The statistic displayed can be the current, live value of a metric, or a function of the value over time, such as an average or sum.
You can use the text panel to display textual information, including hyperlinks to related docs and material that a user of the dashboard might need.
Rows are edited by hovering over the slideout green icon to the left of each row:
To check that MQ statistics are being delivered to Graphite correctly, add a new graph to one of the rows. Grafana will automatically resize the other panels on that row to accommodate it. Click on the title of the new graph and a menu will appear - choose Edit to add data to the graph:
Once in the edit view, add a data series to the graph by selecting "Add Query". Metrics are listed in hierarchies, Selecting the first level in the hierarchy causes a new box to appear where you select the next level in the hierarchy and so on. The first level in the hierarchy is the GraphitePrefix you entered in the collectd configuration file. Clicking on the first "select metric" box in the query should list your Bluemix space ID as an option. Select it, then choose "localhost" as the next level, and "qmgr-<QMNAME>" in the next. The final box should list all of the metrics available for that queue manager:
You can scroll through the list to choose the metric you wish to display. Because the list is repopulated from Graphite every time it may take a while to load. To add multiple data series to the graph, add more queries and select the metric to display for each of them.
singlestat panels are populated in a similar way, choosing the metric you wish to render in the panel.
Once you have edited and configured several panels you can view a snapshot of your queue manager network in a single dashboard:
Note: Make sure you have saved your dashboard using the save icon at the top right of the dashboard.
You can create multiple dashboards and switch between them. Here I've created a dashboard with no graphs on to display some basic system health:
Note: It's possible to show data for more than 1 queue manager by selecting a wildcard when you configure the metric to show, as shown below: