Virtual system patterns fully automate the deployment of complex applications and platforms while taking advantage of best practices, therefore the most important technical role in virtual system pattern development is that of the application deployer.
The application deployer is the subject matter expert on:
- Identifying application prerequisites (both hardware and software).
- Understanding the solution architecture from the perspective of high availability, scalability, failover and fault tolerance.
- Applying best practices for application deployment and understanding the installation and configuration bottlenecks.
- Installing all components of the application.
- Scripting the installation of application (using shell, Jython, DDL scripts).
- Administering prerequisite middleware and software products.
- Running basic functional tests on the application.
Ideally, the application deployer will have enough prior installation, deployment, and configuration experience to identify the automation touchpoints of key manual tasks and to also build into the pattern industry best practices. For example, if most customers or users run with a specific Java™ virtual machine (JVM) heap size in WebSphere®, then this setting should be built into the pattern.
So let's examine some suggestions about the process of planning and designing a virtual system pattern; we'll try to provide helpful commentary at each point. Topics we'll cover include:
- Key concepts in VSP design, such as elasticity, topology, orchestration, and security.
- Some of the important technical aspects when deploying a virtual system pattern:
- Identifying and mapping runtimes to pattern components.
- Incorporating native components into your pattern.
- Reusing existing assets such as scripts and tooling.
- Sizing virtual disks.
- Three ways to extend the functionalities of the base image catalog available in IBM PureApplication System by utilizing script packages, built-in extend/capture tooling, and the Image Construction and Composition Tool (ICON).
- Best practices in the iterative approach to pattern development and testing.
Key pattern design concepts
Review these concepts when designing and developing a virtual system pattern:
Let's discuss each in more detail.
Elasticity in a cloud environment involves automatic horizontal and vertical scaling of your application via dynamic assignment of resources. In a virtual system pattern, WebSphere Application Server environments can be made elastic by using the Intelligent Management Pack (IMP) feature in IBM PureApplication System.
The IMP feature can grow or shrink a WebSphere Application Server cell in a virtual system pattern on demand, based on service level agreements or performance metrics described via policies. An example of how IMP achieves horizontal scaling is when it detects a workload spike in the WebSphere Application Server cell, that might starve out current CPU capacity, it will automatically provision a new WAS node to meet workload demand. Furthermore, IMP is flexible enough to implement vertical scaling when configured. In order to fulfill a response time SLA to prevent performance degradation IMP can trigger the starting of new JVMs in a WebSphere cluster.
If elasticity is a requirement for your application, then consider using the IMP-enhanced WebSphere Application Server environment in IBM PureApplication System.
Figure 1. Selecting IMP enabled WebSphere images in a VSP
If existing topological best practices have been applied within your current environment, these will be relevant to the virtual system pattern as well.
For example, if you use a clustered WebSphere Application Server setup with 8 JVMs and in-memory session replication as your best practice for production, then the same applies to the production flavor of your virtual system pattern.
For a devtest flavor of the virtual system pattern, you may choose a single server configuration and smaller heap sizes on JVMs.
As part of designing the virtual system pattern, it is helpful to create a diagram of the topology in which each product is listed (along with the number of VMs per product) and the relationship between each VM is reflected; for example, if WebSphere Application Server needs to connect to an MQ server, then this communication should be reflected in the topology diagram.
Figure 2. Documenting the topology of the VSP in a diagram
Once a topology has been identified for the virtual system pattern, the next logical step is to list the actions that are needed in each VM to orchestrate the standup of the system. And the order of each action should be determined as well.
Figure 3. Determining order in which topology is orchestrated
For example, if your application's install process requires that a database should be up and running with a schema in place, then it would make sense to orchestrate database setup before the application install process is kicked off.
To enable this kind of orchestration, virtual system patterns allow the designer to specify the order in which virtual machines are brought up and the order in which automation scripts are executed across the virtual machines.
Figure 4. Specifying orchestration order in VSP deployment tooling
It can be helpful to list these orchestration tasks, including their order, in an extended version of the topology diagram since this becomes the blueprint for the virtual system pattern developer.
Security: Directory servers
LDAP support is one of the security-related topics to consider when designing a virtual system pattern. Typically, applications do not mandate dedicated LDAP servers; most apps will connect to an existing LDAP server (such as a corporate LDAP directory) for authorizing access to protected resources. In such a situation, an LDAP server component would not be included in a virtual system pattern.
From a WebSphere Application Server perspective, connection to an existing LDAP server in a virtual system pattern can be captured via a script package that takes LDAP server information (host, user/pwd, etc.) as input parameters. The script package will automate the configuration of an LDAP connection in WebSphere Application Server via a Jython script, thereby alleviating a user of a virtual system pattern to do it manually.
Figure 5. Configuring an LDAP connection in WAS with a script package
If an application requires a dedicated LDAP server, then a new Tivoli® Directory Server instance can first be stood up using the web application Virtual Application Pattern in IBM PureApplication System and WebSphere instances in the virtual system pattern can then connect to the newly stood up Tivoli Directory Server LDAP server — script packages in the virtual system pattern can be used to configure WebSphere Application Server with the new Tivoli Directory Server.
Figure 6. Connecting a WAS instance in a VSP to a TDS server stood up by a virtual application pattern
Gathering input from other key roles in the organization
In some organizations subject matter knowledge can be spread among various roles that might require participation from a broader set of technical experts such as:
- Application architects: This can be the case if a product version migration is being done as part of virtual system pattern creation. For example, if you are upgrading versions of the application server from one level to another by migrating to a virtual system pattern, then an application architect might need to give input on potential code migration as well.
- Application testers: If your application requires complex functional and/or performance testing that is unknown to the application deployer, testers might need to be engaged to validate the proper functioning of the application once deployed within the virtual system pattern.
- Sales engineers and product managers: Employees with insights into your customer's pain points on installation, configuration, life cycle management and scaling can provide valuable input into virtual system pattern design. For example, a product manager might note that a virtual system pattern can potentially boil down a 10-day installation/configuration cycle to 20 minutes, thereby improving the time-to-market or -value. A sales engineer might point out that if a virtual system pattern can stand up a POC environment for a customer in minutes (instead of hours/days), he can reduce the average sales cycle length and demonstrate ease-of-use of your product to your clients.
A recommended best practice is to involve these "resources" in your initial meetings to gather input on design and to discover potential issues up front. As the project moves forward, link the main application deployer to the extended team in case help is needed down the line.
As with any project, non-technical roles are also involved in virtual system pattern development, such as those of project sponsors and project managers. However, descriptions of those roles are outside of the scope of this article.
Identifying and mapping runtimes
You'll definitely want to identify various runtimes in order to effectively map them to virtual system pattern components. Let's look at those processes.
The first step in designing a virtual system pattern is to identify all the runtime components (including exact version numbers) that are needed to host the target application. These typically include:
- Operating systems, including extensions such as specific RPMs on Linux® for example.
- Web servers.
- Application servers.
- Business process servers.
- Messaging and connectivity components such as MQ.
- Custom middleware components such as a custom C++ application server.
The first runtime to check is the operating system; ensure that it is supported on IBM PureApplication System.
Map middleware runtimes to pattern components
Once it has been verified that the required operating system is supported, then the mapping starts:
- IBM middleware components should be mapped to hypervisor editions of middleware products that are shipped with IBM PureApplication System. For example, if WebSphere Application Server V7 on Red Hat Linux is one of the runtime components, then it maps to the WebSphere Application Server V7.X hypervisor edition for Red Hat Linux image that ships with IBM PureApplication System.
- If there is not an exact match of the version number of an IBM product in the IBM PureApplication System image catalog, then an evaluation will need to be done on whether the application can run on a newer version of the product. For example, if the app runs on WebSphere Application Server V188.8.131.52 and the WebSphere Application Server image in IBM PureApplication System is at V184.108.40.206, an evaluation of the application running on the newer version of WebSphere Application Server is needed.
- If the application cannot run on the version level of an IBM product in IBM PureApplication System or if no hypervisor version of the IBM product exists in the IBM PureApplication System image catalog, then the system provides the flexibility to create fully customized virtual images.
Figure 7. Extending and capturing a base image in the catalog
- Custom images: It is possible to create a custom image of the product via the extend/capture capability in IBM PureApplication System or via the ICON tool. In this method, an operating system hypervisor core image is taken as a base and then the product is installed on top of this operating system core image. This customized image is then captured back into IBM PureApplication System for repeatable deployments.
- Support considerations for custom images: Before creating custom images of IBM middleware products, check with IBM support to ensure that such a configuration of the product will be supported.
- If a virtual system pattern component maps to a non-IBM product, then the custom images approach we've just described can be used to include a non-IBM product in the virtual system pattern.
Figure 8. Mapping products in topology to components in VSP
Incorporate native components into a pattern
Native components are products or runtimes that are not operating-system-agnostic. For example, a custom server written in C++ can be considered a native component since it may have been compiled to run on a specific operating system or architecture.
The prime point is to ensure that the native component is compatible with the target hypervisor edition operating system on IBM PureApplication System.
Reusing existing scripts and tooling
Virtual system patterns have been designed to maximize reuse of existing investments that customers may have made in their infrastructure and platform. Scripts that automate the install and configuration of your application will be a key asset to reuse in virtual system pattern development.
Figure 9. A diagram of script reuse
For example, if you have Jython scripts that create a WebSphere Application Server cluster, configure data sources/queue definitions, and install an EAR file, then it will be very easy to reuse such a script in the virtual system pattern environment to orchestrate the stand up of your application. The same applies to DB2® DDL or SQL files that create schemas and populate your tables with initial data; these scripts can also be easily applied to DB2 components in the virtual system pattern.
Additionally, if you have products in your virtual system pattern for which IBM PureApplication System hypervisor images do not exist, then any existing automation tooling for those products will be very useful in the virtual system pattern. For example, silent install tooling for third-party products (as opposed to GUI-driven install shields) can be reused in virtual system patterns to automate standup of non-IBM products.
Disk sizing considerations
Default configurations of hypervisor edition images come with specific disk sizes. For example, WebSphere Application Server profiles are placed in a 2GB virtual disk and the DB2 data virtual disk size is 10GB by default.
If your application requires different disk sizes, then you will need to extend/capture the respective product images in the IBM PureApplication System catalog with larger virtual disks.
A strategy should also be in place for cleaning up or rolling log files and temporary files that can cause virtual disks to fill up.
Extend and capture components in a pattern
If there is a product in your virtual system pattern design for which a hypervisor image does not exist in IBM PureApplication System (these can be non-IBM third-party products or IBM products for which hypervisor image editions do not exist), then there are three methods to capturing such a product/component in the virtual system pattern:
- Using script packages.
- Using the extend/capture method.
- Using the Image Construction and Composition Tool (ICON).
Using script packages
Script packages allow existing or new scripts to be run on VMs that are provisioned by a virtual system pattern. The reason they are called "packages" is that a user uploads a ZIP or TAR file to the appliance which can contain scripts and associated binary files, as well files upon which the script needs to act. For example, if a script package needs to execute a custom Java program, the user can upload a ZIP file that contains a JAR file along with the shell script that will invoke Java code inside that JAR file.
Furthermore, script packages can access built-in environment variables that make them more intelligent about the cloud they are running in. For example, a script package in a WebSphere Application Server VM can look up the hostname of a DB2 VM (that is also part of the same virtual system pattern) in the cloud at deployment time.
Therefore, script packages can also be used to automate the install and configuration of a third-party component on an IBM PureApplication System hypervisor image.
As to the question of what kind of hypervisor image to choose for installing the third-party product, the choices are between Core OS image and any other hypervisor image already in your pattern. Some considerations need to be made to properly answer this question.
If your third-party product can be co-located on an image which is already part of the virtual system pattern, then it can be simpler to just install the product on that existing image. For instance, if you already have a WebSphere Application Server image in your virtual system pattern and the third-party product can be co-located with WebSphere Application Server, then it will be simpler to just add the script package to the WebSphere Application Server VM.
However, if the product needs its own dedicated VM, then you use a Core OS image in your virtual system pattern and add the script package to it. A Core OS image is a blank hypervisor image with only an operating system installed on it. This Core OS image VM will be stood up as part of your pattern just like other regular IBM product VMs and the script package added to this VM will be executed to install, configure, and start up your third-party product.
So, this can be a good approach provided your third-party product binaries are not very large (at the time of writing this article, there is a 2GB limit on script package sizes uploaded from the IBM PureApplication System GUI) and provided that the install/configure of your third-party product is not very complex and can be automated with shell scripts.
Note that script packages that are larger than 2GB in size can be uploaded via the command line interface of the deployment tool. If large files or product binaries are being packaged with the script package, then it is a best practice for the script to access a network file system or central repository where large files are stored.
While script packages allow customization of hypervisor images (via scripts) at virtual system pattern deployment time, the extend/capture method customizes VMs before the virtual system pattern is deployed and makes the customized VM part of the IBM PureApplication System catalog.
If you want to set a standard baseline for a VM in all your patterns, extend/capture can be a good method to do so. For example, if the default disk size of the WebSphere Application Server hypervisor image profiles directory in IBM PureApplication System does not meet your requirements and you want larger disk sizes for WebSphere Application Server profiles, then you can extend the WebSphere Application Server image in the catalog and give more disk space to the WebSphere Application Server profiles directory. The same principle applies to any other standardization you want to do on VMs, such as having specific Linux RPMs to be present on certain VMs in all patterns.
Additionally, this extend/capture approach can be used to also install third-party products on a Core OS or an existing IBM product hypervisor image. In the extend/capture method, the image to extend is first deployed into the cloud and then manual customization (such as third-party product installation) is performed on the VM that is provisioned for the image. Once customization is complete, the state of the deployed VM is captured into the catalog with a new logical name. Then this customized image can be used in any virtual system pattern.
The extend/capture approach is preferable to script packages (for third-party products) when the installation and/or configuration of the product can not be achieved via scripts. For example, if your product is only installable via a GUI which requires human intervention, then the extend/capture approach is a good option. Additionally, in situations where you want fast on-boarding of a product in a virtual system pattern and do not currently have resources to create scripts for automated installation, then this can be a preferable approach as well.
Also, if your third-party product binaries are very large (greater than 2GB), then the extend/capture method is preferable to script packages.
You'll want to note that script packages prevent what is known as "image sprawl"; they allow you to add different "flavors" to a base image. Without script packages, even slightly different configurations of a base image would require you to go through an extend/capture process which would lead to a lot of images in your catalog.
There are some challenges to the extend/capture method.
First, the customization that a user does on the captured VM is manual, therefore it is not easily repeatable unless it is thoroughly documented. For instance, if you support Red Hat and SuSE Linux, then you will have to do this customization twice for both flavors of Linux. Similarly, if you upgrade to a new version of a hypervisor image down the line, you will have to repeat this manual customization.
Second, keep in mind that certain properties of the customized image are dynamic; they change upon every deployment. For example, each instance of an image in the cloud has a dynamically assigned hostname or IP address. If the third-party product configured on the image needs this dynamic information, an associated script package (which runs at image activation time) will have to be used to update this information in the image.
To address some of these challenges, ICON tooling can be used.
Using the Image Construction and Composition Tool
Image Construction and Composition Tool (ICON) is a tool, shipped with IBM PureApplication System, that allows for repeatable customization of a base virtual image. The extend/capture principle explained above also applies to ICON with one major difference: ICON allows for modular customization of an image via a concept known as bundles— this means that customization of the image is inherently documented and repeatable with ICON.
A bundle in ICON represents the software and configuration scripts needed to install/configure that software and runs on a base hypervisor image. Configuration scripts execute as part of image activation, therefore are capable of understanding the cloud context they are running in (such as dynamically picking up the host name of the VM). Multiple bundles can be added to an image in ICON. And bundles can be reused across different images. For example, a bundle for "product X" can be used to customize a Red Hat or a SuSE image.
Similarly, if a newer version of the base image comes along and needs to be customized, existing bundle(s) can be run again on the new base image in an automated and repeatable manner, thereby reducing the maintenance burden of customizing virtual images.
Therefore, ICON, while requiring a greater investment up front, is preferable to the basic extend/capture approach in IBM PureApplication System when multiple operating system support is involved, when repeatability of image customization is a concern, and when images are upgraded frequently.
Table 1. Which tool is best suited for what tasks
|The tool ...||... is best suited for ...|
|Script packages||Installation of small footprint products|
Products with a silent install option that doesn't require human intervention
Post-install customization steps
Ordered orchestration of topology
|Extend/capture||GUI driven install process with no silent install option|
Installation of products with a large footprint
Quicker deployment time for large topologies
|ICON||Streamlined creation of customized virtual images that can be imported into IBM PureApplication System|
Installation of complex non-IBM products
Iterative development and testing
Finally, let's look at an overall philosophy for development and testing of virtual system patterns, one that also works in most IT development efforts.
It can be tempting to take a "big bang adoption" approach to developing the virtual system pattern, one in which you try to automate the standup and configuration of every component in the virtual system pattern in the first go. But this approach can be risky because it can delay the discovery of unforeseen challenges of bringing up the application in the new environment.
This will especially be the case if you happen to be upgrading to a newer or different version of a product by dint of taking a virtual system pattern approach. For example, you may discover that your application requires some tweaking to work on a different version of a product. Or that a third-party component does not function properly on a specific version of an operating system in the hypervisor image.
To prevent late discovery of such issues, it is best to take an iterative approach to virtual system pattern development. This is done by doing a manual dry run of all tasks that you want to automate with script packages:
- Deploy all the VMs in the topology to the cloud without script packages.
- If you already have existing scripts to perform configuration tasks on the VMs, run them manually by uploading them directly to the VM and invoking them by hand. If you don't have scripts yet, then do all the tasks manually (via a GUI or command line) and document any gaps you found in your original virtual system pattern orchestration steps.
- Once all VMs have been configured and the application has been manually brought up, then run some basic tests on the environment to verify the app.
- Once you have verified that the app works in the environment, then you can start creating scripts to automate all the manual steps you performed.
- If you had existing scripts, then all you need to do is to add them to script packages. However, if you did not have existing scripts, then you can first iteratively develop them on the VMs that you manually configured.
- Additionally, there is an iterative approach to testing script packages as well. Script packages can be specified to run automatically (at VM startup or shutdown) or manually from the workload deployment GUI.
- The advantage of running them manually from the GUI is that updates to script packages are picked up upon each invocation. So this allows you to iteratively test your script packages without having to redeploy the entire virtual system pattern on every change made to a script.
We hope this article, combined with the resources at developerWorks and in the IBM PureApplication System Infocenter, can give you a strong start in developing your own virtual system patterns.
We'd like to thank the following for their support and guidance: Kai Young, Catherine C. Diep, Shaun Murakami, Jason Anderson, and Animesh Singh.
- Learn more about IBM PureSystems™.
- Find IBM PureSystems resources on developerWorks
For more on how to perform tasks in the IBM Cloud, visit these resources:
- Up and download files from a Windows instance.
- Install IIS web server on Windows 2008 R2.
- Create an IBM Cloud instance with the Linux command line.
- Create an IBM Cloud instance with the Windows command line.
- Extend your corporate network with the IBM Cloud.
- High availability apps in the IBM Cloud.
- Parameterize cloud images for custom instances on the fly.
- Windows-targeted approaches to IBM Cloud provisioning.
- Deploy products using rapid deployment service.
- Integrate your authentication policy using a proxy.
- Configure the Linux Logical Volume Manager.
- Deploy a complex topology using a deployment utility tool.
- Provision and configure an instance that spans a public and private VLAN.
- Secure IBM Cloud access for Android devices.
- Recover data in IBM SmartCloud Enterprise.
- Secure virtual machine instances in the cloud.
- In the developerWorks cloud developer resources, discover and share knowledge and experience of application and services developers building their projects for cloud deployment.
- Find out how to access IBM SmartCloud Enterprise; developing on SmartCloud is a great way to practice developing for IBM PureSystems.
Get products and technologies
- See the product images available for IBM SmartCloud Enterprise.
- Join a cloud computing group on developerWorks.
- Read all the great cloud blogs on developerWorks.
- Join the developerWorks community, a professional network and unified set of community tools for connecting, sharing, and collaborating.
Dig deeper into Cloud computing on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Experiment with new directions in software development.
Software development in the cloud. Register today to create a project.
Deploy public cloud instances in as few as 5 minutes. Try the SoftLayer public cloud instance for one month.