How-tos

Using Splunk in a Docker Container on Bluemix

Share this post:

As of May 23rd IBM Bluemix Container Service now provides a native Kubernetes operations experience while removing the burden of maintaining master nodes. Kubernetes itself is based on the Docker engine for managing software images and instantiating containers. Get the details.

One of the things the IBM Bluemix platform (based on Cloud Foundry) supports is logging to external providers. Takehiko Amano’s excellent article on Splunk integration with Bluemix describes the solution.

Splunk is a popular platform for log aggregation. It provides an easy way to aggregate logs from multiple sources, providing a way to index and search them in multiple ways. You can find additional details on the Splunk website.

Takehiko’s solution is excellent, but still requires somewhere to deploy Splunk. However, Bluemix itself provides the IBM Containers offerings (based on Docker technology) where Splunk can be run. This isn’t suitable for robust production environments, where you’d want the logging to be externalised from the Bluemix infrastructure, but for “quick ‘n’ dirty” testing, it’s really useful. I’ve documented some steps below that you’ll need to follow to get this up and running with Splunk Light (the simpler, lighterweight edition of Splunk).

Prerequisites

Instructions for this tutorial are written assuming you are using OS X, although they can probably be adapted to other operating systems fairly easily.

Build the Splunk Container

You need to build the Docker container for Splunk locally before pushing it up to the Bluemix containers repository. There’s already a well-established GitHub project for a Splunk Docker container, but we need to add the RFC5424 add-on as per Takehiko’s article to get Splunk to recognize the logging format.

I’ve already forked the GitHub repository and added most of the changes required to do that, but you will need to download the add-on itself first.

  • Open a terminal and clone my repository, checking out the bluemix branch:
    git clone -b bluemix https://github.com/andrewferrier/docker-splunk/
  • Download the RFC 5424 add-on. You’ll need to sign up for a free splunk.com ID if you don’t already have one. Put the .tgz file in the splunklight directory inside your checked-out git repository.
  • Now build the Docker image (which may take a little while):
    cd/splunklight
    docker build -t andrewferrier/splunk:latest-light .

(If you wish, you can substitute your own namespace prefix in place of andrewferrier – as long as you use it consistently below).

Push the Splunk Container up to Bluemix and start running it

First, log into Bluemix and initialize the container runtime:
bx login
bx ic init

You’ll need to specify an organization and space to work within Bluemix.

Next, double-check what your IBM Containers “namespace” is. If you’ve worked with Containers before, you probably already have one specified. You can check it with bx ic namespace-get. If you haven’t, you’ll need to set one with bx ic namespace-set (I use andrewferrier, for example; but you can set it as anything that’s meaningful to you—it will have to be unique across all users using shared Bluemix).

Now, tag your built image to prepare it for upload to the remote registry:

docker tag andrewferrier/splunk:latest-light
registry.ng.bluemix.net/andrewferrier/splunk:latest-light

(Note that the first andrewferrier above is the prefix we specified previously when we built the image. The second is the namespace on Bluemix itself as just discussed. If you want to work with the UK instance of Bluemix, rather than the US one, change all references to .ng. to .eu-gb.)

Now, actually push the image to the remote registry (this may take a little while):

docker push registry.ng.bluemix.net/andrewferrier/splunk:latest-light

Next, we need to create some persistent volumes for both the /opt/splunk/etc and the /opt/splunk/var filesystems within the container:

bx ic volume create splunk-etc
bx ic volume create splunk-var

Start running the container. Notice that we’re exposing two TCP ports, 8000 (which will be used over HTTP to access the Splunk console), and 5140 (which will be used to push syslog messages from Bluemix to Splunk).

bx ic create -m 1024 -p 8000 -p 5140 --env SPLUNK_START_ARGS="--accept-license" --volume splunk-etc:/opt/splunk/etc --volume splunk-var:/opt/splunk/var registry.ng.bluemix.net/andrewferrier/splunk:latest-light

Once the container has started running, the Bluemix CLI will print out the container ID. You typically only need the first few characters—enough to uniquely identify it (e.g. abc1234).

Now check which public IP addresses you have free to assign to the container:

bx ic ips

This should print a list of IPs (probably two if you’re working with a trial Bluemix account)—pick any IP which is not assigned to a container (if you have no unassigned addresses, you’ll either need to pay for more or unbind one from an existing container first). Now bind that IP address to your newly created container:

bx ic ip-bind 1.2.3.4 abc1234

Next, you’ll need to create a user-provided service to stream the logs from your application(s) to Splunk:

bx cf cups splunk -l syslog://1.2.3.4:5140

Setting up a TCP listener within Splunk

Now we need to set up a data listener within Splunk to listen for data on TCP port 5140 (essentially, this is the same procedure as Takehiko’s original article).

Open the Splunk console in a browser using the URL http://1.2.3.4:8000 (obviously, change the IP address for the one you picked above). Log in using the default username/password pair admin/changeme (Splunk will then encourage you to immediately change the password, which you should).

On the home screen, click “Add Data” to add a data source:

Select “Monitor”:

Select “TCP/UDP” to add a TCP-based data listener:

Enter Port 5140 (the same port we exposed from the Splunk Docker container above):

Select rfc5424_syslog as the source type (which corresponds to the Splunk add-on we installed previously). You may find it easiest to type rfc into the dropdown box to select this. Also, you may want to create a new index to index data from Bluemix. In this case, I’ve created one called bluemix:

Review the settings you’ve entered and add the data listener.

Clone and push a demo application

In this article, we’ll clone a sample Node.JS application locally and then push it to Bluemix, so we can bind it to the user-provided service we just defined to use it to test the Splunk integration.

cd <some_temporary_directory>
git clone https://github.com/IBM-Bluemix/get-started-node
cd get-started-node
curl https://new-console.ng.bluemix.net/get-started/docs/manifest.yml > manifest.yml

Now edit manifest.yml it to change name and host to a unique name (e.g. TestAppForSplunkAF (note that this name must be unique within the whole of Bluemix, which is why I use my initials to make this unique).

You also need to modify lines of the server.js file to look like this:

var port = process.env.VCAP_APP_PORT || 8080;

(This ensures that the application will pick up the correct port number from the Bluemix deployed environment).

Now push the application up to Bluemix:

bx cf push

Bind that service to any application you wish:

bx cf bind-service TestAppForSplunkAF splunk

And restage each application:

bx cf restage TestApp

Testing the logging mechanism

Probably, just in the act of re-staging your application, you’ll already have generated some logs. However, to make things a bit more interesting, open the endpoint for your application (e.g. http://testappforsplunk.mybluemix.net/ or similar, modify for the name of your application!) in a browser, and refresh it a few times.

Now, you should start to see your logging information appearing through Splunk. Assuming you set Splunk up as shown above, and created a new non-default index called bluemix, you should simply be able to search for everything in the bluemix index:

You should see some search results appear like this:

Search Results.

Further steps

The world is now your Oyster! You can use any standard Splunk searching mechanism to find logs.

Any questions or comments, please contact me @andrewferrier on Twitter.


Andrew is a Worldwide Bluemix Solution Architect, taking responsibility for end-to-end adoption of IBM Bluemix and providing guidance in the Bluemix Garage. Previously, he led the IBM Cloud Services Mobile Practice in Europe, working with IBM customers on the IBM MobileFirst platform. Andrew has presented extensively on Mobile, Dojo, REST, and Web APIs, contributing Intellectual Capital to the IBM and WebSphere communities, as well as writing two Redbooks, and numerous posts on IBM Mobile Tips ‘n’ Tricks ‘n’ Tricks and SOA Tips ‘n’ Tricks, both of which co-founded. When not in the Garage, Andrew can be found speaking at conferences, such as IBM InterConnect and the European WebSphere Technical Conference.

More How-tos Stories

99.95% availability. Balancing release velocity and reliability

Availability and reliability are rarely at the front of developers minds when delivering new applications on Bluemix. The ease and speed of creating and deploying new features is very seductive.

Continue reading

Deploying to IBM Cloud private with IBM Cloud Developer Tools CLI

IBM Cloud private is an application platform for developing and managing on-premises, containerized applications. It is an integrated environment for managing containers that includes the container orchestrator Kubernetes, a private image repository, a management console, and monitoring frameworks.

Continue reading

Quickly Develop, Build, and Deploy Applications on IBM Cloud with DevOps services – Part 2

In this post, we'll talk about how you can create a DevOps toolchain in Bluemix using Bluemix Continuous Delivery if you've already built and deployed an application.

Continue reading