MQ on OpenStack, part one: Creating an image using Packer
Arthur Barr 120000QMD4 Visits (11917)
This blog entry is the first part in a series on using IBM MQ with OpenStack.
OpenStack is an Infrastructure as a Service (IaaS) which provides a platform and unified API for managing infrastructure resources, including virtual machines, networks and storage. Declaring "support" for OpenStack is not simple, due to the huge variety of underlying hardware and software providers which OpenStack can use. However, IBM MQ currently declares support for many of the individual provider technologies, such as the KVM, Hyper-V and VMWare hypervisors.
The first part of this series describes how to build an OpenStack image with IBM MQ installed, so that you can create virtual machines with MQ already available.
Background - Immutable infrastructure
There are two main approaches to managing virtual machine software. One approach is to use the base images supplied by your cloud provider (which usually just have the base operating system installed), and then entirely manage the software stack using configuration management software such as Chef, Ansible, Puppet or Salt. The other approach is to adopt "imm
Both approaches have advantages and disadvantages, but the advantages of using custom images include the following:
Of course, maintaining (large) VM images is a challenge in itself, but tools like Packer can help, as we'll discuss in this blog entry.
There are a few tools you need to install and configure in order to follow the instructions in the later sections:
This blog entry is going to use the popular open source tool Packer, which allows you to create virtual machine images using a variety of different cloud providers (for example, OpenStack Amazon Web Services and Microsoft Azure). Packer is a command line tool which can be downloaded here.
You will also need access to an OpenStack cloud - we'll be using the command line to manage your cloud environment. Note that there is a new, unified command client, but that doesn't currently support everything we'll be doing in this blog entry, so for now, we'll stick with the more mature proj
pip install python-novaclient
The simplest way to set up your command line environment (for Packer and the OpenStack command line) is to download the OpenStack RC file from the OpenStack dashboard, which can be found under "Compute", then "Access & Security", under the "API Access" tab. This file provides a shell script, which you should run with the "source" command, for example:
The script populates a set of environment variables (for example, OS_USERNAME for your OpenStack user name). Windows users can either set up the environment variables manually, or use an emulation environment such as Cygwin.
Section 1 - Prepare your OpenStack project
Your OpenStack administrator may have pre-configured parts of your OpenStack project for you, or you may have already used the environment for other servers, in which case you may be able to skip to the next section. If however, you have a new, vanilla OpenStack project, there are a few things you need to do before you can start working with it.
In later sections, we're going to use the Packer tool to run our image build process. Packer will create a virtual machine (VM) using a pre-existing image (for example, a clean installation of Linux), connect to that VM over the network, run some "provisioning" scripts, then take a snapshot of the resulting disk, and delete the VM. There are a few things we need to set up first:
There are lots of ways that your OpenStack networking could be set up. However, it is typical for OpenStack administrators to create a "public" or "external" network to represent either the public Internet, or your company's internal network. OpenStack then allows you to create "floating" IP addresses on that network, and allow you to route traffic from those addresses to your virtual machines. This is useful to allow your service (e.g. a queue manager) to retain the same IP address, even if you change which virtual machine is hosting that queue manager. You can look for a pre-existing public network by running the following command:
Once you've identified the network name used in your environment, you can run the following commands, replacing $PUBLIC with the network name in your environment:
neutron net-create build-net
These commands create a new network called "build-net", and add a router to make it possible to route traffic from the public network to this new network. They also creates a floating IP address on the public network. Make a note of the floating IP address, as we'll use it later. Note that we're configuring the subnet to use the Google public DNS nameservers (188.8.131.52 and 184.108.40.206), which you may want to change if you have local alternatives.
Next, we should create a security group. Security groups essentially apply firewall rules to allow or prevent access to your virtual machines. You can create a group as follows:
This creates a virtual firewall which allows ICMP traffic (e.g. for "ping"), and access to port 22 (SSH).
Finally, we should create an SSH key pair, which we'll use later when we create a test virtual machine:
nova keypair-add my-key > ~/.s
This creates a private key (a ".pem" file) and saves the corresponding public key with OpenStack. This is the only copy of your private key, so keep it somewhere safe.
Section 2 - Define a Packer template file
Packer uses a template file to describe what we want in our image, and some options for our cloud provider (in this case, OpenStack). Here's an example template, which you should copy to a file named "mq-
You need to alter this file to work with the settings for your OpenStack environment:
neutron net-show build-net
Section 3 - Create the install script
The Packer template we created in the last step referenced a "provisioner" script. Packer supports a wide variety of provisioning tools, including Chef, Ansible, Puppet, SaltStack. In this blog entry, we'll just use the simple shell script provisioner. See my previous blog entry for how you might be able to use Chef here instead.
Here's an example shell script, which you should put into a file called "install-mq.sh" in the same directory as your Packer JSON template:
# Fail on error
# Recommended: Update all packages to the latest level
# Download and extract the MQ installation files
# Recommended: Create the mqm user ID with a fixed UID and group, so that the
# Configure file limits for the mqm user
# Configure kernel parameters to values suitable for running MQ
# Accept the MQ license
# Install MQ using the RPM packages
# Recommended: Set the default MQ installation (makes the MQ commands available on the PATH)
# Clean up all the downloaded files
# Clean up unwanted files, to help ensure a smaller image file is created
This script updates Linux to the latest packages, installs a few prerequisites, downloads MQ from the location specified in an environment variable (you can change this in your Packer template), accepts the MQ license, installs MQ, then cleans up temporary files.
Section 4 - Build the image
Now you're ready to run the following command to build the image, which will take several minutes to complete.
packer build mq-openstack.json
Once this command complete, you should see an image called ibm-mq has been created. You can check this using the following command:
This image is what we'll be using as the basis for new virtual machines in the next section.
Section 5 - Test the image
You can test your new MQ image by creating a new VM. We'll use the network we used for the build, and the key pair we created in section 1.
Run the following command, making sure to replace details like the flavor and network ID in same way as you did for the Packer template. You also need to replace the $IP variable with the floating IP you created in section 1.
nova boot test-server --image ibm-mq --flavor m1.medium --key-name my-key --nic net-name=build-net --security-group build-security
Now you can SSH into the VM as follows, replacing $IP with your floating IP address:
ssh -i ~/.s
Note that some cloud providers prevent remote login to root in their base images, so you may need to use a different user than "root" here.
Once you've successfully connected, you should be able to run MQ commands as normal.
The steps to actually create the virtual machine image are fairly simple - most of this blog entry has been learning about and configuring your OpenStack environment. If you've followed all the steps, you should now have a reusable image with MQ installed.
In future parts of this series, we'll look at ways you can use this image, to take your next steps towards running MQ on OpenStack. See part two: Mana