The complexity involved in real world software solutions is sometimes daunting. Each server requires a software stack composed of layers of software (operating system, middleware, and applications). The solution is often distributed over multiple machines for scalability and availability. For each solution deployed, the software stack is separately installed and configured directly on each server using traditional deployment techniques.
As discussed in previous developerWorks articles, using virtual images greatly simplifies the deployment process by using pre-installed virtual images of entire solutions. In addition, IBM Tivoli Provisioning Manager provides the capability to store a library of virtual image templates, use metadata to represent configuration data for the virtual images, and represent relationships between the images. By combining the virtual image techniques described in the previous articles with the features available in Tivoli Provisioning Manager, it's possible that you could deploy a sample IBM WebSphere Application Server cluster and IBM DB2® Universal Database in just about ten minutes.
This article combines virtual image techniques with capabilities provided in Tivoli Provisioning Manager to fully automate the deployment of software solutions that span multiple machines. The Tivoli Provisioning Manager Graphical User Interface (GUI) is used to select the solutions for deployment. The sample Tivoli Provisioning Manager workflows provided in the IBM Automation Package for Virtual Software Appliance on the Tivoli Open Process Automation Library (OPAL) are used to drive the deployment and customization of the images.
The steps in this example are organized to correspond with roles that are traditionally separate within an organization. There are four major steps for using Tivoli Provisioning Manager to deploy composite virtual appliances. The first two steps require application knowledge. The last two steps involve Tivoli Provisioning Manager and are common for any composite virtual appliance. The steps are (Figure 1):
Create the composite virtual appliance images.
Creation of the virtual images is independent on whether Tivoli Provisioning Manager will be used for deployment. This step involves understanding common deployment patterns, determining configuration and interdependency information, and actually creating the images.
The Tivoli Provisioning Manager metadata for the composite virtual appliance is created in this step.
Prepare the Tivoli Provisioning Manager deployment environment.
This step involves putting the images and metadata into Tivoli Provisioning Manager, and setting up the host environment.
Deploy and start composite virtual appliance images.
This step performs the deployment using Tivoli Provisioning Manager. The Tivoli Provisioning Manager GUI uses the metadata to display the list of available composite virtual appliances, asks you for customization information, and then the workflows deploy the images to the target environment.
Figure 1. Using Tivoli Provisioning Manager to automate deployment of composite virtual appliances
Details for each of these steps are described in the next sections.
Although using virtual images does not eliminate the complexity of installing and configuring software, it can move the complexity upstream, thereby reducing the number of times the full installation and configuration are performed, and minimizing the number of people exposed to the full installation and configuration process.
In this step, composite virtual images are installed and configured by skilled individuals. This step is done once enabling downstream consumers of the images to use the images without having to install or configure the software to operate together. Since there is investment required to build reusable virtual image templates, the first task is to determine the common deployment patterns.
Determine common usage patterns
A virtual image can be created for virtually any physical deployment, but the most valuable virtual images are those representing deployments that are repeated multiple times, either for different organizations or different applications. Additionally, the more complicated the initial install and configuration, the more value the virtual image(s) will encapsulate.
For example, if you have many IBM WebSphere Application Server or IBM WebSphere Portal application deployments (especially cluster deployments), then creating virtual images can significantly expedite the process. Figure 2 shows a common WebSphere Application Server cluster topology, with the business logic tier including deployment manager and application server managed nodes, and the data persistence tier using DB2. The topology can be distributed over several machines for scalability and availability.
Once you determine the configuration pattern(s) to implement, the next step is to create the virtual images.
Figure 2. Example WebSphere Application Server to DB2 deployment
Create template images
The creation of composite virtual appliances involves capturing the configuration and interdependencies of the solution into a set of virtual machine images. The images can be created by directly installing into virtual images, or captured using physical to virtual (P2V) tooling. The golden images in this example were installed and configured by hand to ensure a full understanding of the image content.
After the images are initially installed and configured, the next step is to analyze the images and convert them into reusable image templates, which can be used to deploy multiple application environments. This step is the most difficult part of the process, requiring a full assessment of what unique configuration values are required for each deployment. The goal is to have as much pre-installed and pre-configured as possible, while keeping the number of images to as few as possible.
Add activation logic
For all images, you will most likely want to be able to change network information (IP address, hostnames, and so on), user information (user IDs and passwords), and remote connectivity (such as database and directory connections). The article Automating deployment and activation of virtual images provides sample scripts for updating IP address and user information for Linux, as well as a framework to insert your software-specific configuration changes.
Beyond the basic network, user, and remote connectivity data, you will also need to decide how much pre-configuration to apply in the image, versus what to do at deploy time. For example, the article Using virtual images to deploy WebSphere Application Server provides a technique where only one WebSphere Application Server virtual image is created. The customization of the image as a deployment manager, standalone, or managed node is done when the image is first activated, based on a deploy time parameter. This technique, as shown in Figure 2, supports using one WebSphere template image to deploy a deployment manager and multiple application nodes.
Identify underlying infrastructure requirements
In addition to pre-installing and pre-configuring virtual images, deploying to a virtual environment provides an opportunity to prescribe specific requirements for the host platform. For example, along with the virtual image template, you should provide the requirements specifications for resources such as the virtual memory and virtual CPU needed for each image to achieve optimal performance.
The expectation is that this information will become part of the Open Virtual Format (OVF) specification. Since the standard is not yet available, the next section describes how to represent the requirements in a virtual server template in the Tivoli Provisioning Manager DataCenter Model (DCM).
Once the virtual images are created, there are a set of steps specific to Tivoli Provisioning Manager that must be performed. The first part of this work, creating the metadata, requires knowledge of the composite virtual appliance. This step is necessary in order to leverage Tivoli Provisioning Manager’s DCM and workflow engine to deploy a virtual appliance. Let’s take a closer look at the required metadata. (The complete sample metadata file for this example is available for download.
Representing the composite virtual appliance
The overall composite virtual appliance is described as a distributed application. A distributed application can be composed of multiple modules; in this case, multiple virtual images. As shown in Listing 1, a distributed application is represented as a software application module in Tivoli Provisioning Manager. Each virtual machine that is contained in the distributed application is specified with an application module entry element. The workflow is also specified here (also shown in Listing 1).
Listing 1. DCM example for software application module
<software-application-module name="WAS Server and DB Composite Appliance" locale="en_US" version="1.0" vendor="IBM" description="This is a WAS Server and DB Composite Appliance" is-draft="false" module-type="APP"> <!-- property definitions need to be added here. --> <application-module-entry software-module="DB Server Appliance" position="1"/> <application-module-entry software-module="WAS Server Cell Appliance" position="2"/> <!-- software-resource-template definition need to be added here. --> <use-workflow workflow-id="Mohegan_Appliance_Deploy" /> </software-application-module>
Representing the individual images
Each composite virtual appliance can contain multiple virtual images. For example, the "WAS Server and DB Composite Appliance," as shown earlier in Figure 2, includes a WebSphere Application Server virtual image and a DB2 virtual image. Each virtual image is specified with a software stack object. Listing 2 shows the "DB2 Server Appliance" image entry. The software stack object contains several property elements for information such as where the virtual machine template is stored in the file repository.
Listing 2. DCM example for software stack
<software-stack name="DB Server Appliance" locale="en_US" description="This is the DB part of the WAS-DB Appliance " stack-type="Declared"> <!-- property definitions need to be added here. --> <!-- software-resource-template definition need to be added here. --> <!-- software-stack-entry definitions need to be added here. --> <!-- installable-package definition need to be added here. --> </software-stack>
Representing the software
The specific software within each virtual image is represented by the software module name metadata. This entry, as shown in Listing 3, includes the underlying infrastructure requirements and the set of parameters to be configured.
Listing 3. DCM example for software module
<software-module name="SUSE_V_10" description="SUSE Enterprise Linux" version="Version 10 Update 1" vendor="Novell" is-draft="false"> <!-- property definitions need to be added here. --> <!-- software-resource-template definition need to be added here. --> </software-module>
Representing the resource requirements and parameters
The resource requirements for each virtual machine are specified with a virtual server template. A virtual server template, as shown in Listing 4, can include several resource requirement elements, such as CPU shares, memory, and hard disk space.
Listing 4. DCM example for Virtual Server Template
<virtual-server-template name="VMware_WAS-Cell_VM"> <resource-requirement resource-type="cpu" how-many="1" size="1.0" is-shared="false"> <property component="KANAHA" name="sched.cpu.max" value="1" /> <property component="KANAHA" name="sched.cpu.shares" value="normal" /> </resource-requirement> <!—additional resource-requirements definitions can be added here. --> </virtual-server-template>
Each software module can have a software resource template, which might include several Template-param elements, which are used to specify the parameters that a user needs to specify for configuration. For example, as shown in Listing 5, the "WAS 6.1 Cell" template has configuration parameters for administrator userid and password. This metadata enables Tivoli Provisioning Manager to collect these parameters in the GUI. The workflows are designed to collect and pass the values to the images for use during activation. One parameter can reference the value of another parameter with the template-param-operand, enabling the expression of interdependencies between the software modules.
Listing 5. DCM example for template parameters
<software-resource-template name="WAS 6.1 Cell Template" software-resource-type= "INSTALLATION" multiplicity-type="Unspecified" software-configuration-type= "Regular" is-selected="true" is-default="true"> <template-param name="secure_adminid" value="wsadmin" parameter-type="String" multiplicity-type="One" is-hidden="false" is-changeable="true" is-encrypted="false"> <template-param-value value="wsadmin" is-default="true" /> </template-param> <template-param name="secure_passwd" value="*****" parameter-type="String" multiplicity-type="One" is-hidden="false" is-changeable="true" is-encrypted="false"> <template-param-value value="pw4was" is-default="true" /> </template-param> <!—additional template-param definitions can be added here. --> </software-resource-template>
Once the metadata file is created, the metadata is imported into Tivoli Provisioning Manager and the workflows are setup as described in the next section.
After the composite virtual appliance images and metadata are created, the Tivoli Provisioning Manager environment must be prepared. This includes copying the images to the repository, importing the metadata, downloading the workflows, and completing any additional set up of the host platforms.
Copy virtual image templates to the file repository
The virtual image templates must be in a Tivoli Provisioning Manager file repository for virtual solution deployment. These file repositories are setup for storing and retrieving large files. The metadata uses the Image.Repository variable to point to the correct file repository, and the Image.Location variable to establish the specific location of the images in the repository. For example, Listing 6 shows the metadata description section for one of the virtual image templates. As shown, the image is kept at images/was61/WAS/was relative to the root of the file repository.
Listing 6. DCM example for image repository
<property component="KANAHA" name="Image.Repository" value="MoheganRepositoryGpfs" /> <property component="KANAHA" name="Image.Location" value="images/was61/WAS/was"/>
Import metadata into Tivoli Provisioning Manager
In addition to copying the image template, the metadata for the image needs to be imported into the DCM. This, however, simply involves running a program:
with the URN of the file as the parameter (the URN for a file starts with
Set up the workflows from OPAL
Once the metadata describing the virtual appliance has been imported into Tivoli Provisioning Manager’s DCM and the virtual image templates are copied into the correct file repository, deploying the virtual appliance becomes a simple task of invoking workflows. A set of workflows for deploying virtual appliances are available on OPAL. Download and install these workflows as described in the package.
These workflows coordinate the deployment of the composite virtual appliance and fork a process to copy each virtual image template and customize it, based the user input and dependency information encoded in the metadata. The workflow samples are written to deploy virtual machines represented as described in this article and should not require changing. However, the workflows are only samples and should be tested before using in your specific target environment.
Set up the host platforms
The images need to be deployed to host platform(s) with the correct hypervisor and resources available. If your Tivoli Provisioning Manager environment is not already set up for the host platforms to be selected, you will need to add them:
- From the Tivoli Provisioning Manager panel, select Add a Host Platform Server, and then enter then name of the host platform (for example,
- Add a network interface and SAP credential.
- Add the hypervisor software and related resources (for example, VMWare ESX Server, CPU, Harddisk Memory, and so on).
(For detailed information on adding a host platform, see your Tivoli Provisioning Manager documentation.)
Additionally, the workflows use SSH on the host platform to store the virtual images. On each of the host platforms, you will need to verify that SSH (Open Secure Shell) is set up such that the user of the host platforms (for example, root@hostplatform1) can ssh login to root and tioadmin of the Tivoli Provisioning Manager server machine (for example, root@tpmserver1 and tioadmin@tpmserver1). Also, ensure that root and tioadmin on the Tivoli Provisioning Manager server machine can ssh login to the users of the host platforms.
- From the Tivoli Provisioning Manager panel, select Add a Host Platform Server, and then enter then name of the host platform (for example,
Once the library of virtual image templates is created in Tivoli Provisioning Manager, deploying a composite virtual appliance becomes a simple process. Since the composite virtual appliance is represented as a distributed application, you interact with the Tivoli Provisioning Manager Distributed Applications panel to select a composite virtual appliance (Figure 3).
Figure 3. Composite virtual appliance selection
Review and customize
After the composite virtual appliance is selected, a panel listing each virtual image template is displayed (Figure 4). For the WebSphere Application Server and DB2 Composite appliance, the deployment manager, WebSphere Application Server, and DB2 images are all displayed. The internal implementation actually maps the deployment manager and the application server to the same image template file.
Figure 4. WebSphere Application Server and DB2 Composite Appliance
Continuing on, a customization panel for Distributed Application Deployment is built from the metadata. As shown in Figure 5, all the customization parameters, such as HostName, DomainName, NodeName, and so on, are displayed. Default values supplied in the metadata will be used if you do not specify some options.
Figure 5. Distributed Application Deployment customization screen
After the customization parameters are entered, deployment is either scheduled or can be started immediately. No further end user interaction is required.
Sit back while Tivoli Provisioning Manager coordinates distributed deployment
The Tivoli Provisioning Manager workflows transparently deploy the virtual appliances. The workflows copy each virtual machine image to the corresponding target machine in parallel. The workflow process includes:
- Checking and deducing the amount of resources (such as CPU shares, memory, and disk space) that are needed by each virtual machine from the total amount of resource available to the hypervisor.
- Copying the respective virtual image template to the corresponding hypervisor.
- Generating a virtual machine configuration file for the virtual image.
- Registering the virtual machine to the hypervisor.
- Registering the software applications contained in each software appliance with Tivoli Provisioning Manager.
The workflows also gather the customization parameters and provide them in activation files to the virtual images. After the virtual images are customized, an "on button’ is provided to enable the coordinated start up workflow to start the solution.
Use the deployed environment
Once the deployment is completed, the information for the deployed virtual appliance is captured by Tivoli Provisioning Manager’s DCM as a distributed deployment. All the information, including where the software is deployed and the configuration parameters, are captured. Thus, any utilities and mechanisms implemented in Tivoli Provisioning Manager for software management can be leveraged to manage the deployed virtual appliances, just like other software deployed in Tivoli Provisioning Manager.
In our lab, we ran an example deployment of the WebSphere Application Server and DB2 composite appliance. We deployed a WebSphere Application Server Network Deployment cluster with one deployment manager, nine application server nodes, one DB2 node, and the Trade6 application. The total deployment completed in less than ten minutes, which is an improvement of several orders of magnitude over the traditional approach, which could take several hours for each machine in the topology.
In our virtual appliance deployment using Tivoli Provisioning Manager, the configuration only required selecting the host platforms and specifying less than ten parameters for each image instantiation. We also further reduced the number of parameters by capturing the interdependency among the set of parameters in the metadata, and only requiring these values to be input one time. The Tivoli Provisioning Manager workflows passed the parameters to multiple images. The workflows, supporting the parallel copying and configuration of pre-captured images, significantly reduces deployment time, compared to constructing physical machines by installing and configuring one layer of software on top of another on each machine.
This article described a radically simplified method for deploying and managing real world software solutions. In this method, the interdependencies in distributed software solutions are captured by virtual machine images, with corresponding metadata describing how the virtual machine images can be configured. A set of Tivoli Provisioning Manager workflows automatically deploy a solution based on the metadata, and significantly reduce the amount of human time needed for deploying complex software solutions.
Automating deployment and activation of virtual images, August 2007
Using virtual image templates to deploy WebSphere Application Server, May 2007
IBM Automation Package for Virtual Software Appliance, October 2007
DMTF Accepts new format for Portable Virtual Machine from Virtualization Leaders, Distributed Management Task Force
Tivoli Provisioning Manager product information
Tivoli Provisioning Manager Information Center
IBM developerWorks Tivoli zone
IBM developerWorks WebSphere Application Server zone
Get products and technologies
Jian Yin is a research staff member at the IBM Watson Research Center. His research interests include scalability, availability, fault tolerance, security, system management, and data mining. He has published over 20 conference and journal papers in the systems area.
Ruth Willenborg is a Senior Technical Staff Member in IBM's WebSphere Technology Institute working on virtualization. Prior to this assignment, Ruth was the manager of the WebSphere Performance team responsible for WebSphere Application Server performance analysis, performance benchmarking and performance tool development. Ruth has over 20 years of experience in software development at IBM. She is co-author of Performance Analysis for Java Web Sites (Addison-Wesley, 2002).
Diana Arroyo is a Sofware Engineer at the IBM T.J. Watson Research Center. She is currently working on integrating virtualization modeling and placement tools with the DMTF-Open Virtual Machine Format (OVF) standard.
Hidayatullah H. Shaikh is a Senior Technical Staff Member and Manager at the IBM T.J. Watson Research Center. He is currently working on various Systems Management service products in IBM Services Research, working closely with IBM Global Technology Services. His areas of interest and expertise include business process modeling and integration, service-oriented architecture, grid computing, e-commerce, enterprise Java, database management systems, high-availability clusters, remote infrastructure management and virtualization. You can contact Hidayatullah at firstname.lastname@example.org.