When I talk with WebSphere CloudBurst users, the topic of custom virtual images comes up frequently. In some cases they simply want to customize a shipped IBM Hypervisor Edition, and in other cases they want to create a completely custom image. Creating a customized version of an IBM Hypervisor Edition is relatively easy since we give you extend & capture in WebSphere CloudBurst. Creating a completely custom image has historically been a bit tougher, mostly owing ot the fact that there was not a standard tool or process for image assembly. I am happy to say that today's publication of the IBM Image Construction and Composition Tool changes all that.
Watch a demo of the IBM Image Construction and Composition Tool
The primary purpose of the Image Construction and Composition Tool is to enable a modular approach to virtual image construction, while taking into account the typical division of responsibilities within an organization. The tool allows the right people within an organization to contribute their specialized knowledge as appropriate to the virtual image creation process. This means OS teams can handle the OS and software teams can handle the appropriate software. A separate image builder can then use both OS and software components to meet the needs of users within the organization. Best of all, the image builder does not need intimate knowledge of how to install or configure any of the components in the image. They simply need to know which OS and software components to use.
When using the Image Construction and Composition Tool, you start by defining the base operating system you wish to use for your images. You can do this by importing an existing virtual image with an OS already installed, providing an ISO for the OS, or pointing to a base OS image on the IBM Cloud. The bottom line is that you have necessary flexibility to start with your certified or ‘golden’ operating system build. Once you have the base OS image defined in the Image Construction and Composition Tool, you can start defining custom software for use in the images you will compose.
In the tool, bundles represent the software you wish to install within a virtual image. The definition of a bundle contains two major parts: Installation and Configuration. The installation component of a bundle tells the Image Construction and Composition Tool how to install your software into the virtual image. You provide a script or set of scripts that install the necessary components into your image, and you direct the tool to call these scripts. These tasks run once during the initial creation of the virtual image, thus allowing you to capture large binaries, long-running installation tasks, or other necessary actions directly into your image.
The configuration section of a bundle defines actions that configure the software installed into the image. Like with the installation tasks, you provide a script or set of scripts for configuration tasks. Unlike installation tasks that run exactly once, configuration scripts become part of the image’s activation framework and as such, run during each image deployment. Using the tool, you can define input parameters for configuration scripts and optionally expose them so that users can provide values for the parameters at image deploy-time. Configuration tasks are important in providing flexibility that allows users to leverage a single virtual image for a number of different deployment scenarios.
Once you have your base OS image and one or more bundles defined in the Image Construction and Composition Tool, you can compose a virtual image. To compose a virtual image, you extend the base OS image and add any number of bundles into the new image. A base OS image plus a set of bundles defines a unique image.
After you define the image you want to construct, you initiate a synchronize action in the Image Construction and Composition Tool. When you start the synchronize action, the tool first creates a virtual machine in either a VMware or IBM Cloud environment (based on how you configured the tool). Next, the installation tasks of each bundle you included in the virtual image run to install the required software. Finally, the tool copies the configuration scripts from each bundle into the virtual machine and adds them to the image’s activation framework. This ensures the automatic invocation of all configuration scripts during subsequent image deployments.
Once the image is in the synchronized state, you can capture it. Capturing the image results in the creation of a virtual image based on the state of the synchronized virtual machine. The tool also automates the generation of metadata that becomes part of the virtual image package. When the capture of the virtual image completes, you can export it from the Image Construction and Composition Tool and deploy it using WebSphere CloudBurst, Tivoli Provisioning Manager, or the IBM Cloud.
I am excited for users to get their hands on the Image Construction and Composition Tool. I believe it represents the first big step in helping users to design and construct more sustainable virtual images. Did I mention it is completely free to download and use? Visit the Image Construction and Composition Tool website for more details and a download link. I look forward to your comments and feedback.
Though I feel like we've come a long way in some of the initial confusion surrounding IBM CloudBurst and WebSphere CloudBurst, I still get quite a few basic questions on the solutions. The two most common questions are, 'Are they different products?', and 'Can/should I use them together?'. I put together a really brief overview that answers these questions and talks about the basics of the combined solution. I hope it provides a good introduction!
One of my favorite books from childhood is If You Give a Mouse a Cookie. Although targeted at children, the book illustrates a frequently occurring human behavior that is important for all of us understand. That behavior is the tendency for escalating expectations. The book offers this up by starting out with the simple action of giving a mouse a cookie. The mouse in turn asks for a glass of milk, various flavors of cookies, and on and on, until the mouse circles back to asking for another cookie.
Nearly all of us exhibit this same kind of behavior, and it can often produce positive results. In particular, in IT we always push for the next best thing or a slightly better outcome. Personally, I am no stranger to this behavior because I experience it from WebSphere CloudBurst users quite frequently. In these cases, it usually revolves around one particular outcome: speed of deployment.
Bar none, users of WebSphere CloudBurst are experiencing unprecedented deployment times for the environments they dispense through the appliance. The fact that we say you can deploy meaningful enterprise application environments in a matter of minutes is far beyond just marketing literature. Our users prove it everyday. However, just because they are deploying things faster than ever does not mean they are content to rest on those achievements. They want to push the envelope, and I love it.
For our users looking to achieve even speedier deployment times, I offer up one reminder and one tip. First, analyze all of your script packages to ensure you are using the right means of customization. If you have some scripts that run for considerably longer than most other script packages, you may want to at least consider applying that customization by creating a custom image. You still need to adhere to the customization principles outlined here, but you may benefit from applying the customization in an image once and avoiding the penalty for applying it during every deployment. You may also be able to break this customization out with a combination of a custom image and script packages. For instance, instead of having a script that installs and configures monitoring agents, you may install the agents in a custom image and configure them during deployment. Being selective about how and when you apply customizations can go a long way in improving your deployment times.
In addition to the reminder above, I also have a tip. Take a look at all of the script packages you use in pattern deployments and look to see if there are any that you can apply in an asynchronous manner. In other words, identify customizations that need to start, but not necessarily complete as part of the deployment process. Going back to our example of configuring monitoring agents during the deployment process, it may be important to kick off the configuration script during deployment, but is it crucial to wait on the results? Maybe not. If it is not, consider defining the executable argument in your script package in a manner that kicks off the execution and proceeds -- i.e. nohup executable command &. This approach can save deployment time in certain situations.
My advice to users of WebSphere CloudBurst: keep pushing your deployment process! Pare as many minutes off the process as you can. I hope that the tips above help in that regard, and be sure to pass along other techniques that you have found helpful.
When you build application environments in WebSphere CloudBurst, there are three main elements that comprise those environments: virtual images, patterns, and script packages. It is likely that at some point you will want to export your environments from a particular WebSphere CloudBurst Appliance. This may be in order to apply version control techniques, share resources among multiple appliances, backup business-critical files, or any number of other reasons. Whatever the reason, WebSphere CloudBurst provides the necessary facilities to support both image and pattern export. WebSphere CloudBurst provides export capability for virtual images that you can access via the web console and CLI. In addition, when you download the CLI from the appliance, you get a sample script called patternToPython.jy that you can use to facilitate pattern export.
The patternToPython.jy sample produces a script that you can use to recreate the targeted pattern on an appliance of your choosing. However, before running the script to recreate the pattern on an appliance, you must ensure that any images and script packages referenced by the pattern exist on the target appliance. Since WebSphere CloudBurst enables you to easily export and import virtual images, all you have to do is account for script packages when attempting to export complete application environments from WebSphere CloudBurst. While the appliance does not directly provide the means to export script packages like it does for images and patterns, the WebSphere CloudBurst Samples Gallery includes a sample that does. You can find this sample in the CLI scripts section of the samples gallery, with the title Export a script package in a portable format.
After downloading the sample CLI script, you simply unzip the archive and use the embedded Jython script from the WebSphere CloudBurst CLI with the following command:
This command will create a ZIP file containing the contents of the script package specified by SCRIPT_PACKAGE_NAME. In addition to simply copying the contents of the specified script package into the new ZIP file, the command will trigger the creation of a cbscript.json file based on the definition of the target script package. This file defines the properties of the script package such as the execution command, command arguments, etc., and the exportScriptPackage.jy script adds it to the newly produced ZIP file.
The result of using this sample is a self-contained ZIP file that you can load into any other WebSphere CloudBurst Appliance. Since the ZIP file includes the cbscript.json file, when you load it into another appliance you do not have to define any of the properties for the script package. This eliminates the potential for definition errors as you move script packages from one appliance to the other and makes it simple to export/import script packages among appliances.
There are a couple of things about the sample worth mentioning. First, if a cbscript.json file exists in the specified script package, the export script will not create a new one. Instead, the sample simply copies the existing one into the new ZIP file. Second, the target script package's contents must be a ZIP file. That is to say, the file associated with the script package in WebSphere CloudBurst must be a ZIP. If you are using anything prior to WebSphere CloudBurst 2.0, this is not an issue since you can only associate ZIP files with script packages. However, WebSphere CloudBurst 2.0 allows you to associate any type of file (ZIP, shell script, python script, etc.) with a script package.
If you are looking to effectively export all of the components of your WebSphere CloudBurst patterns, check out this sample script. I think it will make the process a bit easier for you. As always, comments and feedback are welcome.
When writing a new tool for the WebSphere CloudBurst samples gallery last week, I got the chance to use an API in the CLI that was new to me. Specifically, I got a chance to use the WebSphere CloudBurst CLI in order to retrieve an audit log from the appliance for a specified date period. In case this is new and interesting to you, I thought I would share what I found.
First off, let's take a look at the API I am talking about. It's pretty simple: cloudburst.audit.get(file, start, end). Here, start is the start date for the audit entries and (naturally) end is the end date for those entries. The file parameter simply denotes the location or file object you want to use to store the audit archive retrieved via the get method.
This is a simple enough API. The only wrinkle comes in dealing with calculating the start and end dates. According to the WebSphere CloudBurst Information Center, both the start and end times are 'specified as the number of seconds since midnight, January 1, 1970 UTC. Floating point values can be specified to indicate fractional seconds.' For my use case, I wanted to let a user or calling program pass the start and end times as arguments to the CLI script that retrieves the audit archive. Check out the relevant portion of my script below:
As you can see, the script takes in the start and end time in the MM/dd/yy HH:mm format (i.e. 05/20/10 15:30). It parses the value to produce a date, gets the long value of the date (which is in milliseconds according to the java.util.Date API), and divides that value by 1000. This is to account for the fact that the cloudburst.audit.get method expects you to express the start and end times in seconds. The script passes the converted dates along with the output file location to the get method. The result is a ZIP file that contains an appliance audit, license audit, and PVU audit file for the specified date range.
One of my favorite things about the WebSphere CloudBurst CLI is that it is Jython-based. This means I can leverage Java APIs from my CLI scripts, and that is huge for me because of my existing knowledge of the Java language. You certainly can substitute Python APIs for my use of Java APIs to handle the start and end date calculation. I hope this is helpful, and good luck with the WebSphere CloudBurst CLI!
Since its introduction and initial release around one year ago, activity around WebSphere CloudBurst has been a steady buzz. New images, features, enhancements have been rolling in, and can sometimes be a little overwhelming to digest. With that in mind, I want to address a related and frequent question. What products does IBM support for use in WebSphere CloudBurst?
To answer that question, we only need to look at the IBM Hypervisor Edition images currently provided by IBM. Here's a quick matrix of those images:
One of the most exciting announcements at IBM IMPACT last week was that of the new WebSphere Process Server Hypervisor Edition. This new virtual image allows you to provision complete WebSphere Process Server environments into your on-premise cloud using the WebSphere CloudBurst Appliance. Just like with the other environments you can provision using WebSphere CloudBurst (namely WebSphere Application Server, DB2, and Portal Server), you can stand up these WebSphere Process Server environments in a matter of minutes.
The WebSphere Process Server does not come pre-loaded on the appliance, but it does come with a cool utility that helps you get it on the appliance. The WebSphere Process Server Hypervisor Edition loader provides a wizard-like tool that loads the image into the catalog of an appliance you specify. The tool is simple to use and is included as part of the image package that you download from Passport Advantage.
Not only does the loader above populate the WebSphere Process Server Hypervisor Edition into the appliance's catalog, but it also creates a set of patterns for the WebSphere CloudBurst Appliance. These patterns encapsulate golden topology environments for WebSphere Process Server Hypervisor Edition. At the time of my post, the patterns created by the loader include the following:
Standalone server: This pattern represents a single server instance of WebSphere Process Server. Deployment of the pattern results in a single virtual machine that contains both the server instance and a DB2 instance.
Simulated environment: This pattern contains a single part called a 'Full function control node'. Deployment of the pattern results in the creation of a deployment manager, proxy server, DB2 environment, and three WebSphere Process Server clusters (application target cluster, support cluster, and messaging cluster), all in a single virtual machine.
Scalable environment: This pattern contains a deployment manager, 'Basic function nodes' part, DB2 part, and a proxy server. Deploying the pattern results in the same components as the pattern above, but in this case each component resides in its own virtual machine.
The announcement of the WebSphere Process Server Hypervisor Edition only serves to increase the applicability of WebSphere CloudBurst for constructing on-premise WebSphere clouds. If you have any questions, or want to learn more about this new virtual image, please let me know.
In WebSphere CloudBurst, a script package is your vehicle to provide custom middleware configuration. This may mean installing applications, configuring application dependencies, or otherwise tuning the middleware layer. Script packages are essentially ZIP files that include some executable (shell script, wsadmin script, Java program, etc.), and optionally, artifacts that support the execution of the script. As was the intention, you can achieve just about anything you want with a script package. This allows you to be as flexible and creative as you need to be, but it can also leave you asking "Where do I start?" In this post, I want to take an in-depth look at constructing and using a script package in WebSphere CloudBurst.
Specifically, I want to create a script package that supplies configuration functionality for something I believe a fair number of you do: change the default ports used in WebSphere Application Server. To create this and deploy a pattern using the script package, I do the following:
Create a shell script that configures the desired ports
Add the new script as a WebSphere CloudBurst script package
Create a pattern with the new script package
Deploy the pattern and verify the result
First things first. I create the following shell script that configures the ports:
The script uses documented ANT commands included with the WebSphere Application Server to update the ports based on a starting port number. You will notice the script first sources the /etc/virtualimage.properties file. This file is automatically created by WebSphere CloudBurst on every virtual machine it starts. The file is a key/value file with basic information about the WebSphere cell such as the install root ($WAS_INSTALL_ROOT), the profile name ($PROFILE_NAME), host name ($HOSTNAME), and more. For a full list of the data that WebSphere CloudBurst includes in this file, check out this documentation.
In addition to utilizing the standard set of variables provided by WebSphere CloudBurst, my script above also makes use of the $STARTING_PORT variable. Obviously this variable is not in the standard set. In fact, I define the STARTING_POINT variable when I define my new script package in WebSphere CloudBurst.
First I zip up the shell script above and attach it to the new script package. Next, I tell WebSphere CloudBurst where to unzip the script package on the virtual machine, how to invoke the included script, and the name of any parameters to associate with the script. Once that is done I can use the script package in a new pattern.
For the sake of simplicity here, I create a new pattern by cloning an existing WebSphere Application Server single server pattern. I drag and drop the new Configure ports script package on the single part and end up with the pattern shown below.
Now I am ready to deploy the pattern by clicking the Deploy button. During the deployment process I configure each part in the pattern (in this there is only a single part). I supply configuration information like virtual memory allocation, WebSphere cell name, WebSphere node name, and password information. In addition, I also supply a value for the STARTING_PORT parameter that is part of the Configure ports script package included in the pattern. The value I supply here will get inserted into the /etc/virtualimage.properties file on the virtual machine, and the value's key will be STARTING_PORT.
Once the configuration information is supplied, I click OK on the configuration panel and deployment panel, and WebSphere CloudBurst goes about standing up my virtualized WebSphere cell and running my script to configure the ports for the server instance. When it is done, I login to the WebSphere Application Server administration console to verify my results. To do this, I navigate to the configuration for the single application server instance, and pull up its port definitions.
Based on the results I can see my customizations took effect. I successfully captured my own unique WebSphere environment (in this case with a custom port range) in the form of a pattern. This custom environment can be deployed as many times as I need, in an automated fashion, and I'm guaranteed consistent results each and every time.
I hope this gives you a better idea of what script packages are all about and how they can utilize both WebSphere CloudBurst and user-supplied data that exists in the /etc/virtualimage.properties file of each virtual machine. If you have any questions let me know. I'm on Twitter @damrhein, or you can leave a comment right here.
I'm out at the RSA conference in San Francisco this week, and I'm expecting a lot of good conversations about WebSphere CloudBurst and security. This topic always comes up when I'm out and talking to customers, and I approach it from a few different angles.
First of all, WebSphere CloudBurst enables the creation of on-premise clouds (clouds in your data center). This means that you retain control over the resources that make up and support your cloud, and you have the ability to very tightly secure said resources. Notice that I say "you have the ability". I'm careful to point out that on-premise clouds do not inherently make your environment secure. If you don't already have a robust security strategy in place within your enterprise, then simply moving to a cloud model will not solve much. That being said, if you do have a comprehensive security strategy in place, one built around customized processes and access rights, then on-premise clouds are likely to make much more sense for you.
Moving beyond the opportunity for customized security controls provided by on-premise clouds, WebSphere CloudBurst delivers additional, unique security features. It starts on the outside with the tamper-resistant physical casing. If a malicious user attempts to remove the casing to get to the inner contents, the appliance is put into a dormant state, and it must be sent to IBM to be reset. "So what!" you say. If the user removes the casing and gets to the contents, couldn't they simply read the contents off the flash memory or hard disks directly, or insert them into another WebSphere CloudBurst Appliance and read them from there? Nope. All of the contents stored on the appliance's flash memory and hard disks are encrypted with a private key that cannot be changed and is unique to each and every appliance.
If you are at all familiar with WebSphere CloudBurst, you know that the appliance dispenses and monitors virtual systems running on a collection of hypervisors. Obviously then, the appliance must remotely communicate with the hypervisors. In order to secure this communication, all information between WebSphere CloudBurst and the hypervisors (and vice versa) is encrypted. This encryption is achieved by using an SSL certificate that is exchanged when a hypervisor is defined in WebSphere CloudBurst. This certificate must be accepted by a user, thus preventing rogue hypervisors from being defined in WebSphere CloudBurst.
Finally, WebSphere CloudBurst provides for the definition of users and user groups with varying permissions and resource access rights in the appliance. You don't have to turn over the keys to your cloud kingdom when you add a user to the appliance. You have the capability to define varying permissions (from simply deploying patterns, to creating them, all the way up to administering the cloud and appliance), and you have the ability to control access to resources (patterns, virtual images, script packages, cloud groups, etc.) at a fine-grained level. These two capabilities combine to allow you to control not only what actions a user can take, but also on which resources they can take those actions.
WebSphere CloudBurst was designed with focus on delivering a secure cloud experience, and I think it hit the mark. I'm sure I didn't address all your WebSphere CloudBurst and security related questions. If you have something specific in mind, leave a comment on the blog or reach out to me on Twitter. I'll do my best to address your question.
One of the new features that debuted in WebSphere CloudBurst 1.1 is the ability to resize the disks in a virtual image during the extend and capture (image customization) process. If you remember, the virtual images that exist in the WebSphere CloudBurst catalog are made of multiple virtual disks. In WebSphere CloudBurst 1.0 a default size was used for the virtual disks and this could not be changed, even during the image extension process. To be quite honest we got quite a bit of feedback about this, and so with version 1.1 while default sizes are still provided, you can specify the eventual size of each of the virtual disks during the image extension process.
As an example, consider the WebSphere Application Server Hypervisor Edition virtual image. This image contains four virtual disks: one for the WebSphere Application Server binaries, one for the WebSphere Application Server profiles, one for the IBM HTTP Server, and one for the operating system. The default size of each of these disks in the 18.104.22.168 version of the image is 6GB, 2GB, 1GB, and 12GB respectively, for a total of roughly 21GB. While that may be fine for some, what happens if you are going to be installing various other third-party software packages in the image? You may need more disk space for the operating system's virtual disk. Perhaps your WebSphere applications produce log files of considerable size. In that case you may want to increase the default size of the WebSphere Application Server profiles disk space.
Those scenarios and more are exactly why the resizing capability was added. When you extend the WebSphere Application Server Hypervisor Edition 22.214.171.124 virtual image in WebSphere CloudBurst 1.1, you will be presented the option to resize one or more of the virtual disks:
In the case above the default operating system disk size is bumped up to 16GB from the default 12GB size. Also note that in addition to changing the disk size, you can specify the number of network interfaces for your custom image.
Obviously, when you increase the size of the disks within the virtual image you are also increasing the storage requirements for that image when it is deployed to a hypervisor. Keep this in mind when you are calculating the upper bound capacity of your cloud. If you want to see more about how this feature works, check out this video.