As of May 23rd IBM Bluemix Container Service now provides a native Kubernetes operations experience while removing the burden of maintaining master nodes. Kubernetes itself is based on the Docker engine for managing software images and instantiating containers. Get the details.
Splunk is a popular platform for log aggregation. It provides an easy way to aggregate logs from multiple sources, providing a way to index and search them in multiple ways. You can find additional details on the Splunk website.
Takehiko’s solution is excellent, but still requires somewhere to deploy Splunk. However, Bluemix itself provides the IBM Containers offerings (based on Docker technology) where Splunk can be run. This isn’t suitable for robust production environments, where you’d want the logging to be externalised from the Bluemix infrastructure, but for “quick ‘n’ dirty” testing, it’s really useful. I’ve documented some steps below that you’ll need to follow to get this up and running with Splunk Light (the simpler, lighterweight edition of Splunk).
Instructions for this tutorial are written assuming you are using OS X, although they can probably be adapted to other operating systems fairly easily.
Build the Splunk Container
You need to build the Docker container for Splunk locally before pushing it up to the Bluemix containers repository. There’s already a well-established GitHub project for a Splunk Docker container, but we need to add the RFC5424 add-on as per Takehiko’s article to get Splunk to recognize the logging format.
I’ve already forked the GitHub repository and added most of the changes required to do that, but you will need to download the add-on itself first.
Open a terminal and clone my repository, checking out the bluemix branch: git clone -b bluemix https://github.com/andrewferrier/docker-splunk/
Download the RFC 5424 add-on. You’ll need to sign up for a free splunk.com ID if you don’t already have one. Put the .tgz file in the splunklight directory inside your checked-out git repository.
Now build the Docker image (which may take a little while): cd/splunklight
docker build -t andrewferrier/splunk:latest-light .
(If you wish, you can substitute your own namespace prefix in place of andrewferrier – as long as you use it consistently below).
Push the Splunk Container up to Bluemix and start running it
First, log into Bluemix and initialize the container runtime: bx login
bx ic init
You’ll need to specify an organization and space to work within Bluemix.
Next, double-check what your IBM Containers “namespace” is. If you’ve worked with Containers before, you probably already have one specified. You can check it with bx ic namespace-get. If you haven’t, you’ll need to set one with bx ic namespace-set (I use andrewferrier, for example; but you can set it as anything that’s meaningful to you—it will have to be unique across all users using shared Bluemix).
Now, tag your built image to prepare it for upload to the remote registry:
docker tag andrewferrier/splunk:latest-light
(Note that the first andrewferrier above is the prefix we specified previously when we built the image. The second is the namespace on Bluemix itself as just discussed. If you want to work with the UK instance of Bluemix, rather than the US one, change all references to .ng. to .eu-gb.)
Now, actually push the image to the remote registry (this may take a little while):
Next, we need to create some persistent volumes for both the /opt/splunk/etc and the /opt/splunk/var filesystems within the container:
bx ic volume create splunk-etc
bx ic volume create splunk-var
Start running the container. Notice that we’re exposing two TCP ports, 8000 (which will be used over HTTP to access the Splunk console), and 5140 (which will be used to push syslog messages from Bluemix to Splunk).
Once the container has started running, the Bluemix CLI will print out the container ID. You typically only need the first few characters—enough to uniquely identify it (e.g. abc1234).
Now check which public IP addresses you have free to assign to the container:
bx ic ips
This should print a list of IPs (probably two if you’re working with a trial Bluemix account)—pick any IP which is not assigned to a container (if you have no unassigned addresses, you’ll either need to pay for more or unbind one from an existing container first). Now bind that IP address to your newly created container:
bx ic ip-bind 18.104.22.168 abc1234
Next, you’ll need to create a user-provided service to stream the logs from your application(s) to Splunk:
bx cf cups splunk -l syslog://22.214.171.124:5140
Setting up a TCP listener within Splunk
Now we need to set up a data listener within Splunk to listen for data on TCP port 5140 (essentially, this is the same procedure as Takehiko’s original article).
Open the Splunk console in a browser using the URL http://126.96.36.199:8000 (obviously, change the IP address for the one you picked above). Log in using the default username/password pair admin/changeme (Splunk will then encourage you to immediately change the password, which you should).
On the home screen, click “Add Data” to add a data source:
Select “TCP/UDP” to add a TCP-based data listener:
Enter Port 5140 (the same port we exposed from the Splunk Docker container above):
Select rfc5424_syslog as the source type (which corresponds to the Splunk add-on we installed previously). You may find it easiest to type rfc into the dropdown box to select this. Also, you may want to create a new index to index data from Bluemix. In this case, I’ve created one called bluemix:
Review the settings you’ve entered and add the data listener.
Clone and push a demo application
In this article, we’ll clone a sample Node.JS application locally and then push it to Bluemix, so we can bind it to the user-provided service we just defined to use it to test the Splunk integration.
git clone https://github.com/IBM-Bluemix/get-started-node
curl https://new-console.ng.bluemix.net/get-started/docs/manifest.yml > manifest.yml
Now edit manifest.yml it to change name and host to a unique name (e.g. TestAppForSplunkAF (note that this name must be unique within the whole of Bluemix, which is why I use my initials to make this unique).
You also need to modify lines of the server.js file to look like this:
var port = process.env.VCAP_APP_PORT || 8080;
(This ensures that the application will pick up the correct port number from the Bluemix deployed environment).
Now push the application up to Bluemix:
bx cf push
Bind that service to any application you wish:
bx cf bind-service TestAppForSplunkAF splunk
And restage each application:
bx cf restage TestApp
Testing the logging mechanism
Probably, just in the act of re-staging your application, you’ll already have generated some logs. However, to make things a bit more interesting, open the endpoint for your application (e.g. http://testappforsplunk.mybluemix.net/ or similar, modify for the name of your application!) in a browser, and refresh it a few times.
Now, you should start to see your logging information appearing through Splunk. Assuming you set Splunk up as shown above, and created a new non-default index called bluemix, you should simply be able to search for everything in the bluemix index:
You should see some search results appear like this:
The world is now your Oyster! You can use any standard Splunk searching mechanism to find logs.
Any questions or comments, please contact me @andrewferrier on Twitter.
Andrew is a Worldwide Bluemix Solution Architect, taking responsibility for end-to-end adoption of IBM Bluemix and providing guidance in the Bluemix Garage. Previously, he led the IBM Cloud Services Mobile Practice in Europe, working with IBM customers on the IBM MobileFirst platform. Andrew has presented extensively on Mobile, Dojo, REST, and Web APIs, contributing Intellectual Capital to the IBM and WebSphere communities, as well as writing two Redbooks, and numerous posts on IBM Mobile Tips ‘n’ Tricks ‘n’ Tricks and SOA Tips ‘n’ Tricks, both of which co-founded. When not in the Garage, Andrew can be found speaking at conferences, such as IBM InterConnect and the European WebSphere Technical Conference.
Ready to start working with Kubernetes? Want to build your Kubernetes skills? The five tutorials in this post will teach you everything you need to know about how to manage your containerized apps with Kubernetes.
We are excited to announce the availability of Kubernetes v1.14.1 for your clusters that are running in IBM Cloud Kubernetes Service. IBM Cloud Kubernetes Service continues to be the first public managed Kubernetes service to support the latest upstream versions from the community.