IBM Impact 2011 was a wildly busy week! Customer meetings, entertaining keynotes, informative sessions, and hands-on labs packed the 6 days with more than enough action. I spent a lot of the week presenting sessions and conducting labs for the newly announced IBM Workload Deployer. As one would expect with any new announcement, we got tons of questions about IBM Workload Deployer. While I cannot capture all the questions and their answers here, I will try to cover some of the more prevalent ones below.
Question: What happened to WebSphere CloudBurst?
Answer: The short answer is, it simply went through a rename. WebSphere CloudBurst became IBM Workload Deployer v3.0. The version 3.0 acknowledges this is an evolution of what we started with WebSphere CloudBurst, which was at version 2.0. Why remove WebSphere from the name? The fact that this is now an IBM branded offering is more accurate as it is capable of deploying and managing more than just WebSphere software.
Question: What is new in IBM Workload Deployer?
Answer: While there are many new features that I will be talking about over the coming months, the most prominent new facet is the introduction of workload patterns (also referred to as virtual application patterns). As opposed to topology patterns (traditionally referred to as simply patterns in the WebSphere CloudBurst product), workload patterns raise the level of abstraction to the application level. Instead of focusing on application infrastructure and its configuration as you do with topology patterns, workload patterns allow you to focus on the application and its requirements. When using workload patterns, you provide the application, attach policies that specify functional and non-functional requirements, and deploy. IBM Workload Deployer handles deploying and integration the middleware infrastructure necessary to support the application, and it automatically deploys your application on top of that middleware. In addition, IBM Workload Deployer manages the application runtime in accordance with the policies that you specify in order to provide capabilities such as runtime elasticity.
Question: If I am a current WebSphere CloudBurst user, what does this mean for me?
Answer: Not to worry. You will be able to use all of your WebSphere CloudBurst assets (patterns, scripts, images) in the new IBM Workload Deployer. All of the capabilities previously in WebSphere CloudBurst are present in IBM Workload Deployer (terminology may vary slightly -- topology pattern instead of just pattern for instance). Additionally, we continue to expand on the functionality that you are familiar with from WebSphere CloudBurst. This includes updates for Environment Profiles, new IBM Hypervisor Edition images, new pattern building capabilities, and more. Stay tuned for more information about these new features and for information on how you can move your WebSphere CloudBurst resources to the new IBM Workload Deployer.
Question: How do I choose between using workload and topology patterns?
Answer: There are a number of factors that will lead you to using either workload patterns, topology patterns, or both. The primary decision point will be how much control you really need (not want). When using workload patterns, you sacrifice some customization control over the configuration, integration, and administration of the middleware application environment since the workload pattern and management model abstracts away the 'guts' of the system. Everything about the workload pattern is application-centric. On the other hand, topology patterns give you intimate control over the configuration, integration, and administration of the middleware application environment. As a general rule of thumb, if your application requirements match the capabilities of a workload pattern, that is the way to go as it can greatly reduce complexity and cost associated with deployment and management. If a workload pattern does not meet the needs of your application, topology patterns can still greatly reduce cost and complexity and you can tailor them to fit almost any need. Beyond generalities, there is no hard and fast rule for choosing one over the other. It comes down to understanding your application environment and its needs.
Question: Is IBM Workload Deployer an appliance like WebSphere CloudBurst?
Answer: Yes, it is still an appliance, but an updated one! The new appliance is 2U, and it provides more storage, processing power, and memory. It is still just as easy to setup, but just slightly bigger.
Well, that is all for now, but I will be back many times over the coming months with more information. In the meantime, if you have any questions, please leave them in a comment below.
For the last post in my FAQs Revisited series, I'm going to cheat a little bit. Instead of addressing one particular question, I'm going with a grab bag of a few different questions. These are questions that I get asked quite frequently, but do not demand an entire blog post explanation. Let's get on with it.
Question: Do the new software license management capabilities provided in WebSphere CloudBurst 2.0 depend on ILMT or other supporting components?
Answer: No. The license management features are completely standalone. Of course, you can still take advantage of ILMT (through easy integration in WebSphere CloudBurst I might add) to track licenses in your cloud if you so choose.
Question: Can I deploy a pattern, make changes to my virtual system, and then recapture that as an updated pattern?
Answer: You cannot do this with WebSphere CloudBurst alone, but you can use WebSphere CloudBurst in conjunction with the Rational Automation Framework for WebSphere to do just this. Check out this article (shameless plug alert!).
Question: What if I have an urgent operating system fix to apply before IBM delivers an update to the OS in the Hypervisor Edition image?
Answer: You can either manually apply the fix to the appropriate virtual machines, or you could package up the fix as a custom WebSphere CloudBurst fix, load it into the catalog, and use the appliance to automate the application of said fix.
Question: Can I change the install location for WebSphere Application Server in the virtual image?
Answer: I've just shown you all this really cool, useful, and easy to use stuff, and you worry about install locations? Seriously though, I understand the genesis of this question usually has to do with existing scripts that assume a certain install location for WebSphere Application Server. I certainly do not advocate changing those scripts, but you cannot change the install location for WebSphere Application Server in the images. There is nothing to keep you from creating a symbolic link however.
Question: Once I deploy a pattern, what do I need to do to add more processing capacity (i.e. more application server processes)?
Answer: You have a couple of options here. You can use normal WebSphere administration techniques to add more application servers to an existing node. If that will not work (perhaps a particular node is operating at max capacity), you can use the new dynamic virtual machine operations in WebSphere CloudBurst to add an entirely new node/virtual machine. If you find yourself consistently making these types of adjustments to the runtime environment based on ebb and flow of demand, you may also want to consider the Intelligent Management Pack option for WebSphere Application Server Hypervisor Edition.
I hope this FAQs Revisited series was helpful. Stay tuned for a look at some recent work I did to integrate WebSphere CloudBurst deployments with the new WebSphere DataPower XC10 appliance.
The ability to package custom maintenance packages and upload them as emergency fixes is perhaps a lesser known feature of WebSphere CloudBurst, but nevertheless something that's been around since the product's initial release. This is a powerful feature that allows you to build your own fix packages that you can then apply the same way you would use WebSphere CloudBurst to apply a PAK file or fixpack shipped by IBM.
Since IBM is delivering fixes and updates to all of the contents within WebSphere Application Server Hypervisor Edition virtual images (including the OS and IBM software components), you may wonder why you would even want to create your own maintenance packages. One reason would be if you switched out the SUSE Linux operating system shipped with the VMware ESX based images in favor of your own Red Hat operating system. In that case you would be responsible for maintenance to the operating system, and custom maintenance packages would be of interest to you. Another scenario where these custom maintenance packages come in handy would be if you created your own customized images that include non-shipped third-party software in addition to the software shipped in the images. If at some point you have the need to fix or update this software in a running virtual machine, custom maintenance packages provide you the vehicle with which to do just that.
What do these custom maintenance packages look like? In short, they are simply archives or ZIP files. The contents of the archive are largely decided by you, but there is one piece of metadata that is necessary if you want to use WebSphere CloudBurst to apply the maintenance. A file called service.xml is inserted into the root of the archive and tells WebSphere CloudBurst critical information about the custom fix archive. Here's an example of a service.xml file:
Most notably, this metadata tells WebSphere CloudBurst what module or script to invoke to apply the maintenance (Command, this executable is supplied by you), what image versions the fix is applicable to (ImagePrereqs), and the location of the working directory on the virtual machine (Location). In addition to the service.xml file and the executable, you can package up anything else, such as product binaries, which are needed to successfully apply the fix/upgrade/maintenance.
If you haven't noticed, this is an extremely flexible mechanism and can be used for just about anything. I should point out that you can only apply a given fix once per virtual machine, so it's not good for something that you want to run repeatedly against a given machine (check out user-initiated script packages instead). Also, there is a 512MB size limit on the archives. Keep these restrictions in mind when you are deciding how to use custom maintenance packages. If you are interested in learning a bit more about custom maintenance packages or other maintenance techniques, check out this article I co-authored along with Xiao Xing Liang from the IBM SOA Design Center in the China Development Lab.
May is almost here and that means that IBM IMPACT is right around the corner. Just like years past, IMPACT 2010 will be a great chance to get valuable education and insight into IBM WebSphere software and software from across the IBM software family. If you want to hear how IBM software is leading the march toward a smarter planet, register now.
IMPACT 2010 will be a great chance to hear the WebSphere cloud computing story. There will be multiple sessions on the WebSphere CloudBurst Appliance. These include customer-led sessions, internal adoption stories, overviews, and much more. I'll be there running a hands-on lab and delivering a session that discusses integration between WebSphere CloudBurst and IBM Rational tools. Of course, there is more to WebSphere and cloud computing than WebSphere CloudBurst. We have several other sessions that will detail all of IBM WebSphere's work in the cloud.
If you are interested, I put together a short video discussing some of the sessions on tap for WebSphere and cloud computing at IMPACT 2010. I'd also encourage you to check out the social media site for IBM IMPACT 2010. On that site, you will find tweets, videos, and blogs about the conference. Don't forget to sign up, and I hope to see you in Las Vegas!
-- Dustin Amrhein
If you read some of my entries from time to time, chances are you know that you can use WebSphere CloudBurst to apply interim fixes and fixpacks to your deployed virtual systems. When you choose to apply either a fix or fixpack, WebSphere CloudBurst temporarily stops the virtual system, takes a snapshot of the system (the entire WebSphere cell), applies the fix or upgrade, and then starts the system back up. The result is an updated, running WebSphere cell, and if you need to, you can rollback the virtual system to the previous configuration by simply clicking a button.
In WebSphere CloudBurst 1.0 the application of fixes and upgrades were applied via the web console which made it hard to automate this process. However, in WebSphere CloudBurst 1.1 you can use the command line interface to apply fixes and fixpacks to virtual systems. The appliance still takes the actions I described above, thus the process is still simple, safe, and fast. The only difference is the interface through which you drive the application of the maintenance.
What does it look like? Quite frankly, it's very simple. You can go through all of my virtual systems and apply both fixes and fixpacks with the seven line script below:
for virtualSystem in cloudburst.virtualSystems:
fixes = virtualSystem.findFixes()
if len(fixes) > 0:
upgrades = virtualSystem.findUpgrades()
if len(upgrades) > 0:
You can write this script once, save it as a Jython file, and run it with the CLI's batch mode anytime you want to roll out maintenance to your virtual systems. It's really amazing to me that the above SEVEN lines are capable of rolling out all fixes and all upgrades within your WebSphere CloudBurst catalog to every virtual system the appliance is managing. I can't think of an easier or safer way to automate the deployment of fixes/upgrades to your WebSphere environments.
Let me know if you have any questions. As always you can reach me on Twitter @WebSphereClouds.
Since the IMPACT conference, there has been quite a bit of buzz around the new features, capabilities, and enhancements coming in WebSphere CloudBurst 2.0. In addition to the updates for the appliance, there are some updates to the IBM Hypervisor Edition virtual images as well. In particular, there is one new offering that I want to make sure is getting more than a mere 15 minutes of fame.
What is this new offering that deserves some time in the spotlight? Well, it is the Intelligent Management Pack for the WebSphere Application Server Hypervisor Edition. Still not sure what this is? Simply put, it is an optional add-on to the WebSphere Application Server Hypervisor Edition that enables WebSphere Virtual Enterprise capabilities in the environments dispensed by WebSphere CloudBurst.
If you are not familiar with WebSphere Virtual Enterprise, this still may not mean much yet. Essentially, the use of the Intelligent Management Pack means you create environments that contain capabilities to dynamically manage your application runtime environment. This includes, but is not limited to, the following capabilities:
Dynamic clustering: Create WebSphere Application Server clusters whose membership changes autonomically in order to meet the needs of your applications. You create service level agreements to define the needs for your applications.
Application health monitoring: Monitor the health of your applications by assigning health policies. These policies designate the condition to monitor for (i.e. memory leaks), and they dictate what action to take in case the condition occurs (i.e. restart the server).
Application editioning: Manage multiple versions of your applications and roll out new versions of your applications without incurring downtime. This is essential if you consistently deliver updates to your applications deployed in production environments.
On-Demand routing: Build WebSphere CloudBurst patterns that include On-Demand Router parts. On-Demand Routers are a key component of WebSphere Virtual Enterprise environments and act as an enabler of some of the functionality discussed above.
If you are a user of WebSphere Virtual Enterprise, or otherwise knowledgeable with the product, the Intelligent Management Pack should be pretty familiar to you. When you deploy a pattern built from WebSphere Application Server Hypervisor Edition with the Intelligent Management Pack, you end up with a WebSphere Virtual Enterprise cell. When you log into the administration console, you will see the WebSphere Virtual Enterprise console. You can use any of the features in the normal WebSphere Virtual Enterprise product in the environment created by WebSphere CloudBurst.
Be on the lookout for more information concerning the Intelligent Management Pack. I know there is an article in the works, and we will also be working on some short demos for our YouTube channel. In the meantime, please reach out to me here or on Twitter (@damrhein) with any questions or comments.
I write a lot about WebSphere CloudBurst script packages. Typically, I write about what they are, how to create them, and even provide some samples from time to time. I find that most of the time I'm either writing or talking about script packages from the standpoint that they allow you to automate the delivery of customizations to environments you deploy with WebSphere CloudBurst. More specifically, I usually explain how you can include these script packages in your patterns to ensure that your custom scripts execute as part of every pattern deployment. The truth is, that is not the whole story. In fact, it's only 1/3 of the story.
In WebSphere CloudBurst, when you define a script package you also define its execution mode. The execution mode can be one of three values, and it indicates the invocation time for the script. The default value is at virtual system creation, and that tells WebSphere CloudBurst to automatically invoke the script as part of the deployment process. This seems to be the most commonly used execution mode, and in the original version of WebSphere CloudBurst it was the only available mode (which probably attributes to why I only usually tell 1/3 of the story here). As you may expect, there is a wide range of usage scenarios for this class of script packages including installing applications, activating monitoring agents, registering cells with an externally managed DNS server, and much more.
If you are like me (and many humans), you enjoy and actually expect symmetry. In that regard, it probably comes as no surprise that there is a script package execution mode called at virtual system deletion. As the name indicates, this class of script packages executes as an automatic part of the virtual system deletion process. When a user tells WebSphere CloudBurst to remove a virtual system, before it shuts down the machines in the system, it will run each script package marked to execute at virtual system deletion. Typical use cases for these scripts include removing information about the cell from externally managed DNS servers, freeing up connections with external systems, and other external 'clean up' activities.
So this leaves the final execution mode for script packages, the when I initiate it mode to be precise. This class of script packages executes when explicitly triggered by a user. In the virtual machine detail section for a deployed virtual system, you can see a list of user-initiated script packages for a given machine. There is a start button by each of the user-initiated script packages that allows you to invoke the script when, and as many times as you need to. While these script packages have many different use cases, the most common use case is to deploy application updates. Users build these application update scripts, attach them to a pattern, and invoke them whenever they want to deliver an updated application into their already deployed environment.
WebSphere CloudBurst script packages are one of the main vehicles for delivering your customizations to your cloud environments. The three execution modes mentioned above allow you to determine when the right time to deliver those customizations is.
Looking for a reminder of the difference a year can make? If so, just take a look at the last year or so for the WebSphere CloudBurst product. Since about this time last year, we have seen the release of versions 1.1, 1.1.1, 2.0, and 220.127.116.11, each one bringing their own set of major enhancements and features. Owing to this aggressive pace, it is sometimes easy to miss out on the latest capabilities of the product. For that reason, I wanted to give a brief rundown of some (definitely not all) of the major additions to WebSphere CloudBurst over the past year.
PowerVM and z/VM support: WebSphere CloudBurst 1.1 introduced support for PowerVM (based on Power5 and Power6 systems), and version 1.1.1 introduced support for z/VM. This means that a single WebSphere CloudBurst Appliance can provision to VMware, PowerVM, and z/VM virtualization platforms.
Power7 support: WebSphere CloudBurst 18.104.22.168 introduced support for Power7 systems, thus allowing users to take advantage of the significant enhancements provided by Power7 via WebSphere CloudBurst deployments.
Expansion of the IBM Hypervisor Edition portfolio: The portfolio of images that you can deploy using WebSphere CloudBurst now includes WebSphere Application Server, WebSphere Process Server, WebSphere Portal Server, WebSphere Business Monitor, WebSphere Message Broker, and DB2. In addition to adding new images, we also expanded the platform and operating system support for existing images. For example, you can take advantage of the Red Hat Enterprise Linux OS for WebSphere Application Server Hypervisor Edition, and you can deploy WebSphere Process Server Hypervisor Edition to z/VM infrastructure.
Addition of the Intelligent Management Pack: The Intelligent Management Pack is an optional feature of the WebSphere Application Server Hypervisor Edition that allows you to take advantage of autonomic, policy-driven runtime management capabilities in your deployed environments. This includes the ability to create proactive health policies for your environments, assign SLAs to your applications, manage the update of applications, and more.
License management capabilities: In WebSphere CloudBurst version 2.0 and later, you can make use of license monitoring and management functionality. This allows you to get both point-in-time and historical views of software PVU usage within your cloud, and it allows you to setup policies concerning the usage of PVUs for WebSphere CloudBurst deployments.
Environment profiles: WebSphere CloudBurst provides quite a bit of out-of-the-box deployment automation in terms of selecting hypervisors, assigning IP addresses, and more. However, sometimes you need more control over exactly how this happens. WebSphere CloudBurst 22.214.171.124 introduced environment profiles that you can use to exercise more control over how deployment happens in WebSphere CloudBurst.
In my view, this is quite an impressive list of features delivered within a year's time. I should also reiterate that this is by no means a complete list, but just a selection of some of the major enhancements during this time. If you have any questions about the above additions, or if you have any questions on other features, please let me know.
I was very encouraged by the consistently positive response we got at IMPACT for our WebSphere CloudBurst and Rational Automation Framework for WebSphere (RAFW) integration. I believe there were many reasons for this response: accelerated time to value, decreased investment needs for activities that are not core to your business, lowered barrier of entry for provisioning and configuring WebSphere cells, and much more. While those are certainly all very real and valuable benefits, I also believe that quite a bit of interest in this integrated solution comes from the fact that it is applicable to a number of needs common to you, our WebSphere users.
With that in mind, let's look at some (not all) of the scenarios where WebSphere CloudBurst and RAFW integration can help you:
Create WebSphere CloudBurst patterns that include configuration without scripting: Users love our WebSphere CloudBurst patterns. They really see the value in codifying both the topology and configuration of their application infrastructure. However, some users do not have existing WebSphere configuration scripts and do not have the time and/or resource to invest in creating these scripts. They are looking for a solution that provides not only the provisioning of WebSphere environments but also the configuration of said environments (configuration beyond what our IBM Hypervisor Edition images encapsulate, specifically application-oriented configuration). RAFW provides this capability in the form of 500+ out-of-the-box configuration actions for WebSphere environments. This includes actions to create JDBC resources, create JMS queues, deploy applications, configure application containers, and much, much more. You can create WebSphere CloudBurst patterns that contain a special script package, which points back to a RAFW project containing a set of configuration actions. During deployment, WebSphere CloudBurst will provision your WebSphere environment and then cause the invocation of the specified RAFW project, which in turn runs a set of configuration actions against the provisioned environment. This means you can set up full-blown, ready-to-go application environments with absolutely no user-supplied scripting. In fact, I took this approach to setup a J2EE performance benchmark application, DayTrader 2.0, running on WebSphere Application Server. Those of you familiar with the application know this is not a trivial environment to stand up. Yet, I did it without having to personally write a single line of configuration scripting, and it was all ready to go in around thirty minutes.
Creating WebSphere CloudBurst patterns from existing environments: This comes up all the time. I go through a standard introduction to WebSphere CloudBurst, users see the value, love the patterns-based approach, and immediately want to know how they get their existing environments into the form of a pattern. RAFW, along with the special WebSphere CloudBurst script package, can make this a straightforward and hardened process. You use a capability in RAFW to import the configuration of an existing cell, thereby creating a RAFW environment for that configuration. You then create a WebSphere CloudBurst pattern with a topology congruent to your existing environment, attach the special script package I mentioned earlier, and you are done with the import! When you deploy this pattern, you simply specify the RAFW environment that you created earlier (the one that holds the configuration data for your existing environment) and a RAFW project that will apply the configuration data in that RAFW environment to the WebSphere environment provisioned by WebSphere CloudBurst. The creation of the WebSphere environment, as well as its configuration, happens in a completely automated fashion.
Configure, capture, reuse: There are many situations that may require you to make manual changes to a WebSphere cell after it has been deployed. For example, during performance testing for your application, you might discover that you need to tweak the number of available threads in the web container. As another example, for the first setup of a given application environment, you may want to quickly deploy the cell using WebSphere CloudBurst and then manually install and configure your applications to make sure everything is just right. In either case, it is likely that you want to capture the updated configuration and make sure that any future deployments use those updates. Again, WebSphere CloudBurst and RAFW makes this simple. First, you build a pattern that encapsulates your WebSphere topology (the types and quantity of nodes you want) and attach the special script package mentioned above. For the first deployment, you simply specify the name of the new RAFW environment you want to create. Once the system is up, you log into the WebSphere administration console, make your necessary customizations, and then you use RAFW to import that updated configuration thus updating the initially created RAFW environment. For subsequent deployments, you simply deploy the same pattern, specifying the same RAFW environment as well as a RAFW project, which RAFW automatically created for you during the first deployment. This project applies the configuration (the one you manually established and imported into RAFW) to the WebSphere environment setup by WebSphere CloudBurst.
Configure WebSphere environments across virtual and physical settings: It seems that in many cases our users manage the same WebSphere environment across both virtual and physical settings. For example, they may provision the application environment using WebSphere CloudBurst for everything from development to pre-production, and then for production provision that same environment to a set of physical servers. At least, they try to provision the same environment. In reality, it is tough to reproduce the exact same configuration once you break from the WebSphere CloudBurst patterns-based approach. However, if you stored the configuration of your WebSphere cell as a RAFW environment, you could apply that configuration data to a WebSphere cell regardless of whether it existed in the physical or virtual world. Once you move to physical, you do lose out on the fast provisioning, WebSphere intelligence, cloud management capabilities, and automated integration with RAFW that you get when using WebSphere CloudBurst, but if it is in your process to move to physical hardware at some point, reusing the same RAFW environment certainly eases the migration task.
I hope this sheds some light on some of the common issues WebSphere CloudBurst and the Rational Automation Framework for WebSphere can combine to solve really well. This is by no means an exhaustive list, but really meant to point out the broad application of the solution. If you want to see how it works, check out this video.
When it comes to administration of WebSphere environments, I (and many others) am a big fan of scripting. In my view, any administrative action you carry out with frequency > 1 is ideally suited for a script. The downside to not using scripts (longer configuration times, inconsistent configurations, isolated expertise) is simply too steep in most cases. I also realize that simply saying that you should script is not enough. For some, the learning curve can be a bit daunting. Quite frequently, I talk about our samples gallery or provide posts with embedded scripts in the hopes that I can help flatten out this curve a bit.
While these samples can certainly help to speed up your scripting efforts for certain use cases, they are more or less helpful for solving tactical challenges when scripting. If you and your company are embarking down a strategic path that includes beefing up your administrative scripting capability, I would strongly suggest you look at a resource a few of my colleagues pointed me at recently.
The resource I am talking about is the wsadminlib.py package referenced here. This python script file is a collection of hundreds of methods that carry out common WebSphere Application Server administrative tasks. The authors carefully constructed these methods with clear method and parameter names. The result is a script resource that can become the foundation for your custom-crafted administrative scripts.
I recently downloaded the wsadminlib.py script and began constructing WebSphere CloudBurst script packages to utilize it. To say I am impressed would be an understatement. This file makes so many tasks so incredibly simple. Take for instance the creation of an SIBus. That's just a simple call like the following:
wsadminlib.createSIBus(clusterName, nodeName, serverName, SIBusName, scope, secure)
How about associating a shared library with an application or application module? Another one-line call:
wsadminlib.associateSharedLibrary (libName, appName, warName)
Or what about setting a custom property in the webcontainer? You guessed it. One-line:
wsadminlib.setWebContainerCustomProperty(nodeName, serverName, propName, propValue
This is just an extremely small sample of what the wsadminlib.py includes. As I mentioned earlier, there are hundreds of other methods that carry out various tasks including: installing applications, creating core groups, creating virtual hosts, installing BLAs, creating JMS queues, and much more. If you are looking to beef up your WebSphere Application Server scripting efforts, or if you are just starting, I strongly encourage you to look into and make use of this valuable resource!
The reason I suggest the application proxy approach is twofold. First, it affords you the ability of having custom interactions with the REST API. For instance, you may insert logic into the server-side proxy code that returns only a subset of the JSON data contained in the response from the appliance. Alternatively, in an effort to reduce the chattiness on your client-side, you may join JSON data from multiple different REST requests to the appliance to fulfill a single client request. You may even decide to represent the data in an all together different format than JSON. All of these options and many more are available to you if you implement an application-based proxy to the REST API.
The second reason I suggest the application approach is that it is easier, and seemingly safer, to not deal with user passwords on the client-side. If you setup your application proxy, you can configure it to retrieve the appropriate password from a secure location (like an encoded file) based on information passed along in the request. This means the password information is only present in the request (in encoded form of course) from the application proxy to the WebSphere CloudBurst Appliance.
The good news about the application-based proxy approach is that it is simple to put in place. I composed one using the open source Apache Wink project. The Apache Wink project is an open source implementation of the JAX-RS specification (and then some), and it enables you to develop POJOs that are in turn exposed in a RESTful manner. In my case, I had a single resource POJO:
The Apache Wink runtime routes any HTTP GET request whose path is like /resources/* to the getResources method in the WCAResource class. This method passes along information taken from the query string (the host name of the target WebSphere CloudBurst Appliance and the requesting WebSphere CloudBurst username), as well as the HTTP path information and sends it on to the getResource method declared as follows:
The getResource method above uses the WebSphere CloudBurst host name and the request path to construct the URL for the corresponding WebSphere CloudBurst REST API call. Next, it constructs an Apache Wink Resource object and sends the REST request along to the WebSphere CloudBurst Appliance. How do we authenticate this request? We use the WebSphere CloudBurst username (sent as a query string parameter) to retrieve the appropriate encoded password information. Once we have that, we construct the necessary header for basic authorization over SSL.
The application-based proxy shown here is simply a pass-through. It does not manipulate the data returned from the WebSphere CloudBurst REST API, nor does it map a single client-side call to multiple REST requests. However, it would be simple enough to extend it to do any of those things. If you have any questions about the code here, please let me know. I'd be happy to share more of the code, or talk about how and where to extend it.
I’m going to take a different approach this week in the blog. Instead of me telling you about some of the features or uses of WebSphere CloudBurst, I thought I would catch up with someone using the product everyday, WebSphere Test Architect Robbie Minshall. Robbie is responsible for a team of testers that harness a lab of over 2,000 physical machines to put our WebSphere Application Server product through some pretty rigorous testing. Toward the beginning of this year Robbie’s team started to leverage the WebSphere CloudBurst Appliance in order to create the WebSphere Application Server environments needed for their testing.
Robbie, can you tell us a little bit about what the WebSphere Application Server test efforts entail?
In WebSphere Application Server development and test we have two primary scenarios. The first is making sure that developers have rapid access to code, test cases and server topologies so that they can write code, test cases and then execute test scenarios on meaningful topologies. The second scenario is an automated daily regression where in response to a build, we provision a massive amount of WebSphere Application Server topologies and execute our automated regression tests.
Previously we have supported these scenarios through the deployment of the Tivoli Provisioning Manager for operating system provisioning, some applications for checking out environments, and then a lot of automation scripts for the silent install and configuration of WebSphere Application Server cells.
Given those scenarios and the existing solution, what are your motivations for setting up a private cloud using WebSphere CloudBurst Appliance?
We are supporting these scenarios through a pretty complicated combination of technologies. These include silent WAS install scripts, wsadmin configuration scripts, a custom hardware leasing application and the utilization of Tivoli Provisioning Manager for OS Provisioning. This solution is working very well for us though as always we are looking for areas to improve, opportunities to simplify and to reduce our dependency on investment in our custom automation scripts. Mainly, there were 3 areas where we wanted to improve our framework: Availability, Utilization and Management. This is why we started looking to the WebSphere CloudBurst Appliance.
Can you expand a bit on what you are looking for in those three areas?
The first focus area we have is availability of environments. We really wanted to lower the entry requirement for the skills and education necessary to get a development or test environment. Setting up these environments has just been too hard, too time consuming, and too error prone. Using WebSphere CloudBurst we can provide an easy push button solution for developers to get on-demand access to the topologies they need.
The second area we are looking for significant improvements on is hardware utilization. Our budgets are tight and in our native automation pools we are only using between 6-12% of the available physical resources. In order to improve this we were looking at leveraging virtualization. WebSphere CloudBurst offers the classic benefit of virtualization with the nice additions of optimized WebSphere Application Server placement and really good topology and pattern management. In our initial experiments we were able to push the hardware utilization up to 90% of physical capacity and consistently were leveraging around 70% of our physical capacity.
Finally we are looking to improve and simplify our management of physical resources and automation. We work in a lot of small agile teams and organizational priorities change from iteration to iteration. Not only does WebSphere CloudBurst allow us to maintain a catalog of topologies or patterns for releases but it also allows us to adjust physical resource allocation to teams through the use of sub clouds or cloud groups.
Basically we felt that WebSphere CloudBurst would improve the availability of application environments, enhance automation, and improve hardware utilization all with very low physical and administrative costs.
What were some of the challenges involved with getting a cloud up and running in your test department?
One of our challenges seems like it would be common to many scenarios, especially in today’s world. Our budget for new hardware to build out our cloud infrastructure was initially very limited. Most cloud infrastructure designs depict very ideal hardware scenarios including SANs, large multicore machines, and private and public networks within a dedicated lab. Quite frankly we did not have the budget to create this from scratch. It was important for us to demonstrate value and data to warrant future investment in dedicated infrastructure. After some performance comparisons we were very happily surprised to see that we could leverage our existing mixed hardware within a distributed cloud. The performance of application environments dispensed by WebSphere CloudBurst on many small existing boxes in comparison to large multicore machines with a SAN was very comparable. This allows us to leverage existing hardware, with minimal investment all the while demonstrating the value and efficiencies of cloud computing. That data in turn has allowed us to obtain new dedicated hardware to iteratively build up a larger lab specifically for use with WebSphere CloudBurst.
Specifically with WebSphere CloudBurst, are there any tips/hints you would offer users getting started with the appliance?
Sure. First, we quickly realized as we added hypervisors to our WebSphere CloudBurst setup it was critical to have someone with network knowledge on hand. This is because the hypervisors came from various sections of our lab, and we really needed people with knowledge of how the network operated in those different sections. Once we had the right people we were able to setup WebSphere CloudBurst and deploy patterns within an hour and a half.
Moving forward we continued to have challenges as we dynamically moved systems between our native hardware pool and our cloud. Occasionally the WebSphere CloudBurst administrator would move a system into the cloud but incorrectly configure the network or storage information. This lead to some misconfigured hypervisors polluting our cloud. We overcame this, quite simply and satisfactorily I may add, by creating some simple WebSphere CloudBurst CLI scripts which add the hypervisors, test them individually, by carrying out a small deployment to that hypervisor, and then move the correctly configured hypervisors into the cloud after verifying success. Misconfigured hypervisors go into a pool for problem determination. This has allowed us to maintain a clean cloud, and we are able to dynamically move our hardware in and out of the cloud to meet our business objectives.
We also use the WebSphere CloudBurst CLI to prime the cloud so to speak. Before using a given hypervisor in our cloud, we execute scripts that ensure each unique virtual image in our catalog has been deployed to each of our hypervisors at least once. When the image is first deployed to a hypervisor, a cache is created on the hypervisor side of the connection, thus meaning subsequent deployments do not require the entire image to be transferred over the wire. This gives us consistent and fast deployment times once we are using a hypervisor in our cloud.
I would assume that like many applications deployed on WebSphere Application Server, your team’s applications have several external dependencies. Some of these dependencies won’t necessarily be in the cloud, so how did you handle this?
You’re right about the external dependencies. Our applications and test cases run on the WebSphere Application Server but are dependent upon many external resources such as databases, LDAP servers, external web services etc. WebSphere CloudBurst allows us to deploy WAS topologies in a very dynamic and configurable way but the 1.0.1 version does not allow us to deploy these external resources in the same manner. This was overcome by using script packages in our patterns. These script packages allow us to associate our test applications with various patterns we have defined. The script package definition also allows us to pass in parameters to the execution of our scripts. We supply these parameter values during deploy time, and these values are used to convey the name or location of various external resources. The scripts that install our applications can access these values and ensure the application is properly integrated with the set of resources not managed by the appliance.
What is your team looking to do next with WebSphere CloudBurst and their private cloud?
The next challenge on our plate is to keep up with the demand of our expanding cloud and to develop a more dynamic relationship between our native pools and our cloud using the Tivoli Provisioning Manager. These are fun challenges to have and we look forward to sharing our progress.
I'm glad I got to spend some time with Robbie to glean some insight into their work and progress with WebSphere CloudBurst. I hope this information was useful to you. It's always nice to hear about a product from practitioners who can give you hints, tips, gotchas, and other useful information. Be sure to let me know if you have any questions about what Robbie and his team are doing with WebSphere CloudBurst.
A while back I co-authored an article along with Chris Ahl from Tivoli and Ken Klingensmith from WebSphere Technical Sales about the customization of virtual images in WebSphere CloudBurst. In the article we approached image customization as a means to enable IBM Tivoli Monitoring for the operating system within virtual machines dispensed by WebSphere CloudBurst. Today I posted a short demonstration that discusses and shows this particular integration scenario. If you are interested, but haven't had time to read the article, you may want to watch the video first as it should give you a good overview of the process and results.
Talk of Tivoli reminds me that IBM Pulse 2010 is just around the corner. I'll be going to discuss WebSphere CloudBurst and how it can be paired with software from IBM Tivoli for high-value integration scenarios. In the session I'll be talking about the Tivoli Monitoring integration as well as other key points such as our integration with Tivoli Service Automation Manager, IBM CloudBurst, and more. The best part about the session is that I will be co-presenting alongside a WebSphere CloudBurst customer that will dole out practical advice for using WebSphere CloudBurst within the enterprise. Join us on Tuesday February 23rd from 3:30 - 4:30 in Conference Center 306.
Remember, any time you have questions about WebSphere CloudBurst please pass them along. You can leave comments on this blog, or you can reach me at my new Twitter location @damrhein.
In WebSphere CloudBurst, a script package is your vehicle to provide custom middleware configuration. This may mean installing applications, configuring application dependencies, or otherwise tuning the middleware layer. Script packages are essentially ZIP files that include some executable (shell script, wsadmin script, Java program, etc.), and optionally, artifacts that support the execution of the script. As was the intention, you can achieve just about anything you want with a script package. This allows you to be as flexible and creative as you need to be, but it can also leave you asking "Where do I start?" In this post, I want to take an in-depth look at constructing and using a script package in WebSphere CloudBurst.
Specifically, I want to create a script package that supplies configuration functionality for something I believe a fair number of you do: change the default ports used in WebSphere Application Server. To create this and deploy a pattern using the script package, I do the following:
Create a shell script that configures the desired ports
Add the new script as a WebSphere CloudBurst script package
Create a pattern with the new script package
Deploy the pattern and verify the result
First things first. I create the following shell script that configures the ports:
The script uses documented ANT commands included with the WebSphere Application Server to update the ports based on a starting port number. You will notice the script first sources the /etc/virtualimage.properties file. This file is automatically created by WebSphere CloudBurst on every virtual machine it starts. The file is a key/value file with basic information about the WebSphere cell such as the install root ($WAS_INSTALL_ROOT), the profile name ($PROFILE_NAME), host name ($HOSTNAME), and more. For a full list of the data that WebSphere CloudBurst includes in this file, check out this documentation.
In addition to utilizing the standard set of variables provided by WebSphere CloudBurst, my script above also makes use of the $STARTING_PORT variable. Obviously this variable is not in the standard set. In fact, I define the STARTING_POINT variable when I define my new script package in WebSphere CloudBurst.
First I zip up the shell script above and attach it to the new script package. Next, I tell WebSphere CloudBurst where to unzip the script package on the virtual machine, how to invoke the included script, and the name of any parameters to associate with the script. Once that is done I can use the script package in a new pattern.
For the sake of simplicity here, I create a new pattern by cloning an existing WebSphere Application Server single server pattern. I drag and drop the new Configure ports script package on the single part and end up with the pattern shown below.
Now I am ready to deploy the pattern by clicking the Deploy button. During the deployment process I configure each part in the pattern (in this there is only a single part). I supply configuration information like virtual memory allocation, WebSphere cell name, WebSphere node name, and password information. In addition, I also supply a value for the STARTING_PORT parameter that is part of the Configure ports script package included in the pattern. The value I supply here will get inserted into the /etc/virtualimage.properties file on the virtual machine, and the value's key will be STARTING_PORT.
Once the configuration information is supplied, I click OK on the configuration panel and deployment panel, and WebSphere CloudBurst goes about standing up my virtualized WebSphere cell and running my script to configure the ports for the server instance. When it is done, I login to the WebSphere Application Server administration console to verify my results. To do this, I navigate to the configuration for the single application server instance, and pull up its port definitions.
Based on the results I can see my customizations took effect. I successfully captured my own unique WebSphere environment (in this case with a custom port range) in the form of a pattern. This custom environment can be deployed as many times as I need, in an automated fashion, and I'm guaranteed consistent results each and every time.
I hope this gives you a better idea of what script packages are all about and how they can utilize both WebSphere CloudBurst and user-supplied data that exists in the /etc/virtualimage.properties file of each virtual machine. If you have any questions let me know. I'm on Twitter @damrhein, or you can leave a comment right here.
If you've read anything I've written about WebSphere CloudBurst up to this point you know all about patterns. Using the appliance you can easily and quickly build, deploy, and manage these representations of your middleware application environments. Today, I want to focus in on the deployment piece in particular and take a look at how you can easily automate this process.
You can use the WebSphere CloudBurst web console to deploy patterns, and when doing so you can even schedule the deployment to happen at a later date. This scheduling capability certainly gets you on the road to an automated deployment process, but what if you want to take it one step further and eliminate the need for someone to login and manually move around the web console to schedule automated deployments? In this case, you can use either the CLI or the REST interface that WebSphere CloudBurst offers.
In this post I thought I'd take a look at using the CLI interface in order to set the stage for some nice automation around pattern deployment. It starts out with a properties file that provides details about my deployment. This includes the cloud to deploy to, the pattern to deploy, password information, and the time at which the virtual system should start.
SYSTEM_NAME_PREFIX=New App Development
TARGET_CLOUD=Default ESX group
TARGET_PATTERN=WebSphere single server
Imagine that the properties file above gets written as the result of some other action, such as the completion of your application's build process. With the properties file in place, and I'll point out that your properties file can and probably will be more robust than above, let's move on to the code that handles the deployment process based on the information in said file. First, we have a small amount of CLI code to retrieve and parse the input data (I omitted the straight-forward properties retrieval for space):
from datetime import datetime, timedelta
from java.util import Properties
from java.io import FileInputStream
// read in and retrieve properties using java.util.Properties API (i.e. props.getProperty('DEPLOYMENT_DATE'))
parsedParts = deploymentDate.split(" ")
systemName = systemName + "_" + deploymentDate
dateParts = parsedParts.split("/")
timeParts = parsedParts.split(":")
monthPart = int(dateParts)
dayPart = int(dateParts)
yearPart = int(dateParts)
hourPart = int(timeParts)
minutePart = int(timeParts)
Next is the code that actually schedules the pattern deployment:
First we get the desired deployment time and current time as datetime objects. After that, assuming the desired deployment time has not already elapsed, we calculate the difference between the desired deployment time and current time. This difference, in seconds, is then added to the result of the time.time() value to come up with a start time. After that is done, we simply retrieve the cloud that was indicated in the properties file, and then we call the runInCloud method for the pattern indicated. When calling the runInCloud method we supply the name of the virtual system that will be created, password information, and the start time we calculated earlier. As a result of this method call, a task will be generated in the target WebSphere CloudBurst Appliance and the virtual system will be started at the specified time. This will happen in an automated fashion with no human intervention required.
That's really all there is to automating the pattern deployment process using the CLI. In a more complete, end-to-end scenario you may envision the completion of one process, such as an application build process mentioned above, result in the writing of the properties file and in turn the call into the CLI to deploy a pattern. As always, feel free to send me any comments or questions.