One goal in cloud computing is to enable a rich selection of working resources to cloud consumers; another goal is to keep the support tasks at a manageable level for the teams of image and software developers. It might seem that these two goals are exclusive of each other, but this series (see series overview) demonstrates how effective use and management of software bundles and image resources can make both of these objectives attainable.

This article:

• Provides an overview of software bundle and image management concepts.
• Explains the resource model.
• Compares this model to other software package management platforms.
• Discusses use cases.
• Surveys the available tools.

The concepts are illustrated with the use of an example energy-saving calculator application. Learn how to implement a simple solution, including a very small amount of Python code and instructions for building an RPM package. Read how to deploy that software package to a virtual machine on the cloud. Subsequent articles in the series will investigate individual tools, APIs, and use cases in more detail, using the same example application.

This article is suitable for cloud users and planners wanting to know how to efficiently manage cloud resources in a sustainable way. No knowledge of programming is necessary. Although some very short script examples are provided, all the necessary background is explained. The examples use Linux® but can easily be adapted to work on Windows®.

Some context to start

In the context of IBM SmartCloud Enterprise, a software bundle is a collection of software installation files, configuration files, and metadata that can be deployed on a virtual machine instance.

You can create and manage software bundles in SmartCloud Enterprise using the IBM Image Construction and Composition Tool (ICCT), the SmartCloud Enterprise REST APIs, and third-party tools.

Software in virtual machine images is usually pre-installed and configured by the image developer. A problem with this "freeze-dried" approach is that you may not be able to find an image with the combination of software that you want and this can lead you to the do-it-yourself approach where you choose an off-the-shelf virtual machine images and consider it as a base to install your own software with. This can lead to a proliferation of virtual machine images, known in IT circles as virtual machine image sprawl.

Another problem is that virtual machine images designed for specific use cases tend not to be very usable for scenarios other than the one intended by the image developer. SmartCloud Enterprise provides a bridge that crosses the gap between the freeze-dried approach and the do-it-yourself approach by using virtual machine parameters, activation scripts, and software bundles to create template images on SmartCloud Enterprise.

Another, more portable, solution is the IBM Image Construction and Composition Tool (ICCT), a standalone tool that can add software to virtual machines. ICCT is now included with IBM Workload Deployer 3.1, IBM SmartCloud Provisioning 1.2, and IBM SmartCloud Enterprise. ICCT stores images as Open Virtual Format Archive (OVA) packages that can be deployed using IBM Workload Deployer 3.1, IBM SmartCloud Enterprise, and VMware products.

Virtual machine images can contain:

• Operating systems
• Middleware
• Applications
• User-specific configuration

As you add more to an image it becomes less reusable; the less reusable an individual image is, the more likely you will end up with more images. And tangentially (but nonetheless an important concept), the less you include in an image, the less useful your image is to cloud users.

Deciding what to include in an image and how to make it a template are two fundamental tasks in the art of image development.

Back to top

SmartCloud Enterprise resource model

A simplified logical resource model is shown in Figure1.

Descriptions of the resources, attributes, and relationships are in Table 1.

Back to top

Software package management systems

Managing software packages is a critical aspect of computing systems management, including cloud systems management. Package management is special in a cloud context because you interact with virtual machine instances that are dynamically provisioned. This allows an opportunity for provisioning additional software on the fly and also raises users' expectations that the system will "just work."

Cloud software package management systems are similar in concept to and can wrap package management systems that are bundled with operating systems and some software development platforms.

This section outlines Linux software package management systems and apply that to cloud systems. It walks through an example that creates a software module in Python, bundles it as an RPM, and injects it into a cloud-based virtual machine at provisioning time.

RPM and YUM package management

Red Hat Package Manager (RPM) was developed by Red Hat in the 1990s and is a part of the Linux Standard Base. It has also been ported to other operating systems such as IBM AIX®:

• It helps maintain consistency between packages
• Can be used to obtain information about installed packages
• Verifies packages for authenticity using cryptographic signatures
• Updates already installed packages
• Can retrieve packages from the Internet

There are several front ends for RPM, including Yellow Dog Updater Modified (YUM), up2date, Zypper, and Smart Package Manager. RPM is primarily used on Red Hat Enterprise Linux, CentOS, and Fedora. For Debian-based systems, including SUSE and Ubuntu, Debian package management is used.

YUM was developed at Duke University as a front end to RPM to manage Red Hat Linux systems. YUM provides the additional functions of automatic update and dependency management to RPM and works with central software package repositories. While RPM understands dependencies it will not resolve them. This is a problem where YUM helps out considerably.

To install a package with the rpm command using the -i option:

# rpm -i <package-name>

If there are required dependencies then the rpm command will fail. You must have root authority to install, remove, or update packages.

To install a package with YUM use the yum command with the install option. For example, to install dos2unix:

# yum install dos2unix

If there are dependencies, then YUM will install them automatically.

Debian package management

Advanced Packaging Tool (APT) is the package management system used by Debian systems, including SUSE and Ubuntu. Just as for RPM and YUM, package data is stored in a central database.

You can install a package with the apt-get install command. To install the dos2unix package:

# apt-get install dos2unix

apt-get finds the package information in the remote repository, determines the dependencies, downloads the package, and installs it.

Python modules

Python has built-in features for distribution and installation of modules. From the point of view of installing and consuming Python software, these features are easily used by downloading a module, unzipping it, and executing the setup command. Python is a convenient language for us to demonstrate the software distribution and management process of the example software. You do not need to know anything about Python to understand this article beyond what is explained in this section.

Before we start, let's list some basic Python code reuse terms in Table 2.

To install a Python module:

# python setup.py install

This is normally done as root and installs the module in the directory {INSTALL_ROOT}//site-packages, for example, /usr/lib/python2.6/site-packages. Alternatively, you can install it as a regular user in your user home directory tree with the command:

$ python setup.py install --user

Python provides the Distutils toolkit to help module authors distribute their software that make these commands possible.

Back to top

The example code: A Python distribution module

Let's create a Python module, an installation script, and test it by installing it. The package is an energy calculator called ecalc and contains the single module energy_calculator with the class EnergyCalculator. In this simple example, the code just prints out some sample data; however, it is easy to apply the same concepts to a more complex scenario that requires a highly organized software package management system.

The code for the EnergyCalculator class is:

# energy_calculator.py
"""
energy_calculator module for computing energy usage
"""

class EnergyCalculator:
    """EnergyCalculator class"""

    def __init__(self):
        self._base = 200

    def energy_used(self):
        return self._base

To invoke it from the energy calculator client:

#!/usr/bin/python
# energy_client.py

""" EnergyCalculator client """

from energy_calculator import EnergyCalculator

ec = EnergyCalculator()
print("You used {0} kJ today".format(ec.energy_used()))

When invoked, the example output is:

$ ./energy_client.py
You used 200 kJ today

First, create the setup script:

# setup.py
# Setup script for libenery

from distutils.core import setup
setup(name='libenergy',
      version='0.1',
      py_modules=['energy_calculator'],
      )

Next, create the distribution module:

$ python setup.py sdist

The setup script command in the above example creates a source distribution package. It creates a dist directory with a compressed file libenergy-0.1.tar.gz that can be used to install the package. It does not include the client script energy_client.py. It warns of missing information about the author, URL, and README. The command also creates a manifest in the file MANIFEST that is included in the TAR file. To copy the file libenergy-0.1.tar.gz to your target machine, use the secure copy command:

$ scp -i ${key} libenergy-0.1.tar.gz idcuser@${vm}:libenergy-0.1.tar.gz

The command assumes that you are working on Linux. Windows is similar but needs the scp command installed. It is included with OpenSSH. This uses the SSH key ${key} to copy the file to the virtual machine with host name or IP address ${vm} as idcuser. Define shell variables for these to avoid retyping the values. Then you can unbundle and install it remotely with this command executed from the dist directory:

$ ssh -i ${key} idcuser@${vm} 'gunzip libenergy-0.1.tar.gz; 
 tar -xvf libenergy-0.1.tar; cd libenergy-0.1; sudo python setup.py install'

This SSH script executes remotely to decompress the bundle and install it as root with the sudo command. The remote commands are enclosed in single quotes and separated by semi-colons.

Now that you have installed the energy_calculator module on the target machine, you can use the client program to call it with this script:

$ scp -i ${key} energy_client.py idcuser@${vm}:energy_client.py
$ ssh -i ${key} idcuser@${vm} './energy_client.py'
You used 200 kJ today

These commands first copy and then execute the client script. Execute the commands from the directory containing the client script.

Let's recap what you have achieved at this point:

• You wrote an example Python module and a client script to exercise it.
• You created an installation bundle for the module.
• You installed the module remotely on a virtual machine using SSH.
• You invoked the module remotely on the virtual machine using the client script.

Not bad, especially since you are writing and consuming "your" own software in this example so far. However, the example is simple; there are a few shortcomings that are becoming apparent:

1. The consumer has to know how to install Python modules. We would like to be able to use the same method to install software packages written in other languages too.
2. It is not convenient for people other than Python programmers to find out about it and perhaps update it.
3. It does not describe its prerequisites, such as what kind and level of operating system and Python are needed.
4. To avoid the image sprawl problem, we would like to install it from the cloud console and inject the package into virtual machines. At least, have options other than writing complex-looking bash scripts, such as in the example provided.

Optimally, you should be able to inject the energy calculator module into a virtual machine at provisioning time or when you need to update it automatically.

Back to top

Installing the Python module with RPM

Python has some additional tools to help you create install bundles for Red Hat Package Management (RPM), Windows, and other formats that can help solve shortcomings 1, 2, and 3 we just mentioned. These formats are a more convenient form to use when writing activation and start-up scripts.

This section shows how to write an activation script to test if the energy calculator is already installed. If it is not installed then you can install with an rpm command.

To create an RPM package:

$ python setup.py bdist_rpm

This command creates an RPM package libenergy-0.1-1.noarch.rpm in the dist directory. Copy it to the target virtual machine with an scp command:

$ scp -i ${key} libenergy-0.1-1.noarch.rpm idcuser@${vm}:/tmp/libenergy-0.1-1.noarch.rpm

Install the energy calculator module:

$ sudo rpm -i /tmp/libenergy-0.1-1.noarch.rpm

The rpm command checks whether it is installed or not and does not re-install it if it is run a second time.

Check that the module is installed by running the energy calculator client, energy_client.py, as shown. You will use this RPM file and command next to inject the energy calculator module into virtual machines when they are provisioned in the cloud.

Back to top

Cloning an image using the SmartCloud web portal

The manual process for creating software bundles is not as easy as using tools like ICCT, but it does demonstrate the software bundle process in more detail and is amenable to automation using the SmartCloud Enterprise APIs. For those reasons we will discuss it first.

One of the first steps that is needed is to create an image of which you have ownership. This is needed to edit the metadata and add files that will be injected into virtual machines that will be provisioned based on the image. There are two ways to do this:

1. Start up a virtual machine instance and make the customizations that you want, including installing software.
2. Clone an existing image.

Image cloning is described because it is the most amenable approach to automating and building tools for image management.

To clone an image:

1. Click the Add Image button on the Control Panel | Images page. The Base images window opens.
   
   Figure 2. Cloning an image in the SmartCloud web portal
   
   
   
2. Click the Next button. Step 2 of the wizard opens.
   
   Figure 3. Step 2 of the Clone Image Wizard
   
   
   
3. Enter a name and a description for the cloned image. Verify the data in Step 3 of the wizard and click the Next button.
4. Agree to the terms and conditions in Step 4 and click the Submit button. The image takes a few minutes to clone and appear in the Control Panel and the Asset Catalog.

The contents of the image asset in the asset catalog look similar to Figure 4.

The files include parameters.xml which describes input parameters and files modeling the topology of the image. The files that you are most interested in are in the activation_scripts folder, as shown in Figure 5.

Back to top

Inject installation scripts to VMs during provisioning

The contents of the cloned image from the previous sections in the asset catalog look like this:

The files include parameters.xml which describes input parameters and files modeling the topology of the image. The files that you are most interested in are in the activation_scripts folder, as shown in Figure 7.

The key to injection of files into a virtual machine at provisioning time is the file scripts.txt. This is the scripts.txt file from the cloned RHEL 6 image:

cloud-startup3.txt=/etc/init.d/cloud-startup3.sh
activate.txt=/etc/cloud/activate.sh

This leads to the file cloud-startup3.txt being injected to the virtual machine as /etc/init.d/cloud-startup3.sh and the file activate.txt as /etc/cloud/activate.sh.

To inject your own files into a virtual machine:

1. Download the files in the activation_scripts folder, placing them in a folder of the same name.
2. Add the libenergy-0.1-1.noarch.rpm file to the folder.
3. Create the script libenergy-activate.sh:

   /usr/bin/logger 'installing libenergy-0.1' rpm -i /tmp/libenergy-0.1-1.noarch.rpm 2> /tmp/energy.log /usr/bin/logger < /tmp/energy.log

   
   
4. Be careful of using UNIX-style line separators or running the dos2unix utility. You want this script to be copied to the virtual machine and executed at startup. To do that, add it to the activation_scripts folder. Edit the file scripts.txt file as shown:

   cloud-startup3.txt=/etc/init.d/cloud-startup3.sh activate.txt=/etc/cloud/activate.sh libenergy-0.1-1.noarch.rpm=/tmp/libenergy-0.1-1.noarch.rpm libenergy-activate.sh=/home/idcuser/libenergy-activate.sh

   
   
5. Invoke the script from the cloud startup script cloud-startup3.sh. Edit cloud-startup3.sh and add the lines in bold as shown:

   start) echo "== Cloud Starting" if [ ! -e /etc/cloud/idcuser_pw_randomized ]; then echo "Randomizing idcuser password" echo idcuser:'< /dev/urandom tr -dc _A-Z-a-z-0-9 |head -c16' | /usr/sbin/chpasswd touch /etc/cloud/idcuser_pw_randomized fi /sbin/restorecon -R -v /home/idcuser/.ssh chmod +x /home/idcuser/libenergy-activate.sh /home/idcuser/libenergy-activate.sh ;;

   
   
6. Compress the folder and call the resulting file activation_scripts. Edit the image asset for the cloned image and upload the ZIP file. It is decompressed during the upload process.

To test the cloned and modified image, provision a virtual machine based on the cloned image; check that the module is installed by running the energy calculator client, energy_client.py, as shown:

$ scp -i ${key} energy_client.py idcuser@${vm}:energy_client.py
$ ssh -i ${key} idcuser@${vm} './energy_client.py'
You used 200 kJ today

If you do not see the output, You used 200 kJ today then check the message log file /var/log/messages for errors.

By this point, you have learned how to inject a software module into virtual machine without installing a running virtual machine. You only started up the virtual machine to verify the result. This is an important leap in the ability to automate image management in the cloud and is the basis for several tools that are available to help users manage their virtual machine images.

The following sections show how to use some of these tools and how to automate more effectively using APIs. Also, notice that you did not create an independent software bundle resource in the cloud; you directly merged the RPM installation file and the install script with the virtual image metadata in the asset repository. We will discuss cloud-based software bundle resources.

Back to top

Creating composable software bundles

To create a composable software bundle, download and modify one of the templates provided in the SmartCloud Asset Catalog. In the Asset Catalog, find the search link in the left column called Composable Software Bundles (Figure 8). Browse the search results returned.

There are many different bundles available to use as templates. Use the Asset Manager Clone button to clone and modify the bundle. This helps you create resources that you can use in your own automation tools to share between virtual machine images, but they will only be visible to you. To create more widely reusable software bundles, you need to register them using ICCT or another tool.

Back to top

Using ICCT to manage software bundles

ICCT is included in the IBM SmartCloud Enterprise image catalog which enables you to run it on the cloud. To use it go to the SmartCloud Enterprise Control Panel:

1. Select Instance and click the Add Instance button.
2. Select a data center, select the ICCT image from the catalog, and provision a new instance using the provisioning wizard.
3. Enter the ICCT user name and password. After the virtual machine instance is provisioned, you can use the link in the Getting Started page to login.
4. After logging in to ICCT, click the Build and Manage Software Bundles link.
5. On the left panel is a bundle called Enablement Bundle for SmartCloud Enterprise. You should extend this to create your own software bundles if you plan to use ICCT or the SmartCloud Enterprise software bundle APIs.

Back to top

In conclusion

In this article, you've gained a broad understanding of the image and software bundle capabilities in IBM SmartCloud Enterprise by walking through a simple example. Future articles will build on this example using tools that have additional capabilities to build more complex versions of this example and automate many of the manual tasks that were performed here.

Resources

Learn

• For more information on the technologies mentioned in this article:
  • Python documentation
  • Learn Linux, 101: RPM and YUM Package Management
  • RPM Package Manager online
  • Fedora (Red Hat) RPM Guide online
  • Learn Linux, 101: Debian Package Management
  • Establish a system to build custom virtual cloud images (ICCT tool)
  
  
• For more on how to perform tasks in IBM SmartCloud Enterprise, visit these resources:
  • Up and download files from a Windows instance.
  • Install IIS web server on Windows 2008 R2.
  • Create an IBM Cloud instance with the Linux command line.
  • Create an IBM Cloud instance with the Windows command line.
  • Extend your corporate network with the IBM Cloud.
  • High availability apps in the IBM Cloud.
  • Parameterize cloud images for custom instances on the fly.
  • Windows-targeted approaches to IBM Cloud provisioning.
  • Deploy products using rapid deployment service.
  • Integrate your authentication policy using a proxy.
  • Configure the Linux Logical Volume Manager.
  • Deploy a complex topology using a deployment utility tool.
  • Provision and configure an instance that spans a public and private VLAN.
  • Secure IBM Cloud access for Android devices.
  • Recover data in IBM SmartCloud Enterprise.
  • Secure virtual machine instances in the cloud.
  
  
• In the developerWorks cloud developer resources, discover and share knowledge and experience of application and services developers building their projects for cloud deployment.
  
  
• Find out how to access IBM SmartCloud Enterprise.
  
  

Get products and technologies

• See the product images available for IBM SmartCloud Enterprise.
  
  

Discuss

• Join a cloud computing group on developerWorks.
  
  
• Read all the great cloud blogs on developerWorks.
  
  
• Join the developerWorks community, a professional network and unified set of community tools for connecting, sharing, and collaborating.
  
  

About the authors

Alex Amies is a senior software engineer in the IBM GTS Development Lab in the China development lab. He is currently an architect working on the design of the IBM SmartCloud Enterprise. Previously, he acted as an architect and a developer on cloud and security products in other groups within IBM.

Alexei is a member of the SmartCloud Enterprise design team focussing on software bundles, ICON, VPN/VLAN for the cloud business support systems. He is based in the United States.

Andrew is a member of the SmartCloud Enterprise design team focussing on business support systems. He is based in the United States.

Shaikh is an IBM Master Inventor and technical lead of the Business Support Systems for IBM SmartCloud Enterprise and Enterprise+ offerings, based in the United States.

Qiang Guo Tong joined IBM in 2004. He is one of the IBM SmartCloud Enterprise development team leaders responsible for creation of new features. He is also the globalization enablement solution architect of SmartCloud. His technical expertise is both globalization and also JavaEE technologies. He has over 8 years web application development experience based on the JavaEE architecture and focusing on web 2.0 technologies such as Dojo, JSON-RPC.

Randy is a Master Inventor and member of the SmartCloud Enterprise design team. He currently focuses on business support services with specialties in the areas of billing and rating, business process enablement and BSS data model design. Based in Research Triangle Park, NC Randy has broad knowledge across the cloud BSS technology built on a foundation from a long and varied past in the system management arena since joining IBM in 1989.

Scott Peddle is an advisory software developer on the IBM SmartCloud Enterprise. Scott leads projects focusing on integration of Rational Asset Manager within a cloud environment. Scott's experience includes leading a project for integration of WebSphere BPM tools, such as WebSphere Business Modeler and WebSphere Integration Developer via Rational Asset Manager. Scott also was a release engineer for Rational design and construction products, such as Rational Software Architect and Rational Application Developer.

Rate this article

Comments

Back to top

Table of contents

• Some context to start
• SmartCloud Enterprise resource model
• Software package management systems
• The example code: A Python distribution module
• Installing the Python module with RPM
• Cloning an image using the SmartCloud web portal
• Inject installation scripts to VMs during provisioning
• Creating composable software bundles
• Using ICCT to manage software bundles
• In conclusion
• Resources
• About the authors
• Comments

Next steps from IBM

Learn more, join the cloud community

---

• Community: Cloud computing conversations led by IBM community members
• Buy: Try IBM software on IBM SmartCloud Enterprise

Dig deeper into Cloud computing on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers