A while back I had a four part FAQ series inspired by questions arising from customer visits discussing the first release of WebSphere CloudBurst. With the recent release of WebSphere CloudBurst 2.0, I think it is a good time to revisit an FAQ series with an entirely new set of questions.
For the first part of the series, I want to address a question we get all the time now: "What is the difference between WebSphere CloudBurst and WebSphere Virtual Enterprise?" This question was always fairly common, but now even more so because the new Intelligent Management Pack option for WebSphere Application Server Hypervisor Edition allows you to deploy WebSphere Virtual Enterprise cells using WebSphere CloudBurst.
Fundamentally, the difference between the WebSphere CloudBurst Appliance and WebSphere Virtual Enterprise is a complementary one. WebSphere CloudBurst provides a means to create your application environments, deploy them into a shared, cloud environment, and then manage them over time. In this respect, the appliance focuses on bringing cloud-like capabilities to the application infrastructure layer of your application environments. WebSphere CloudBurst does give you management capabilities for your running, virtualized application environments (i.e. applying maintenances and fixes), but for the most part those capabilities do not extend into the application runtime environment.
Now, you may ask why WebSphere CloudBurst does not extend its reach into the application runtime. The answer is simple: We already have a solution that does just that, WebSphere Virtual Enterprise. WebSphere Virtual Enterprise provides capabilities that allow you to dynamically and autonomically manage your application runtime. You can use WebSphere Virtual Enterprise to not only assign performance goals to your applications, but also to declare the importance of a given application meeting its goals relative to other applications in your organization. This enables the dynamic management of your applications and their resources such that your applications perform according to their goals and relative importance to your business. Simply put, you get an elastic runtime at the application layer of your application environments.
As I said, WebSphere CloudBurst and WebSphere Virtual Enterprise are complementary solutions. Both enable notions of cloud computing, but at different layers of your application environments. WebSphere CloudBurst hones in on the application infrastructure components, while WebSphere Virtual Enterprise zeros in on the applications running in those environments. The new Intelligent Management Pack for WebSphere Application Server Hypervisor Edition means that WebSphere CloudBurst can now dispense WebSphere Virtual Enterprise environments into your on-premise cloud. That means you can take advantage of these complementary solutions from a single and simple management plane.
I hope this helps to clear things up. As always, questions and comments are welcome!
For the next installment of this series of FAQs, let's move from product positioning and integration, square into the land of operational procedure. For this post, we will consider you are getting ready to deploy a pattern based on the WebSphere Application Server Hypervisor Edition. During the deployment process, you provide configuration information, which includes a password for a user named virtuser.
You read the documentation, and you understand that virtuser is both an operating system user and the user that WebSphere CloudBurst configures as the primary administrative user for WebSphere Application Server. Naturally, this user owns the WebSphere Application Server processes that run in the virtual machine. While it is convenient that this is all pre-configured for you, you want to know one thing: "Can I define a user besides virtuser?"
It certainly would not be the first time this question came up. The short answer to this is yes, but there are of course caveats. You can define another user and have that user own the WebSphere Application Server processes, but you cannot completely remove the virtuser user, nor should you remove virtuser as the primary administrative user. The reason for this is that WebSphere CloudBurst relies on virtuser when it carries out certain actions such as applying maintenance, applying fixes, or otherwise interacting with the WebSphere Application Server environment.
All that being said, I recently put together a script package that allows you to utilize a user other than virtuser. I hope to put the script package in our samples gallery soon, but here's a basic overview of using the script package and what it does:
Attach the script package to all parts in a pattern that contain a WebSphere Application Server process.
Deploy the pattern and provide the necessary parameter values. These include the name of the new user, a password, a common name, and a surname. The last two bits are necessary when creating a new administrative user in WebSphere Application Server.
During deployment, the script package first creates a new OS user with the specified password.
The script adds the new user to the existing OS users group.
The script creates a new WebSphere Application Server user with the same username and password and grants administrative privileges to the user.
The script shuts down the WebSphere Application Server processes.
The script changes the runAsUser value for all servers to the empty string and sets the runAsGroup value for those servers to users. This allows members of the OS users group to start the WebSphere Application Server process.
The script starts the WebSphere Application Server processes.
There are a few other activities in the script, but that should give you a basic overview. Again, note that the script does not remove the virtuser user or change that user's OS or WebSphere Application Server permissions in anyway. I would also point out that if you use WebSphere CloudBurst to apply maintenance to the WebSphere Application Server environment, it will do so as virtuser and it will restart processes as virtuser, so plan accordingly.
I hope this sheds some light on a very common question. I hope to get the sample up soon, and as always let me know if you have any questions.
One of my favorite books from childhood is If You Give a Mouse a Cookie. Although targeted at children, the book illustrates a frequently occurring human behavior that is important for all of us understand. That behavior is the tendency for escalating expectations. The book offers this up by starting out with the simple action of giving a mouse a cookie. The mouse in turn asks for a glass of milk, various flavors of cookies, and on and on, until the mouse circles back to asking for another cookie.
Nearly all of us exhibit this same kind of behavior, and it can often produce positive results. In particular, in IT we always push for the next best thing or a slightly better outcome. Personally, I am no stranger to this behavior because I experience it from WebSphere CloudBurst users quite frequently. In these cases, it usually revolves around one particular outcome: speed of deployment.
Bar none, users of WebSphere CloudBurst are experiencing unprecedented deployment times for the environments they dispense through the appliance. The fact that we say you can deploy meaningful enterprise application environments in a matter of minutes is far beyond just marketing literature. Our users prove it everyday. However, just because they are deploying things faster than ever does not mean they are content to rest on those achievements. They want to push the envelope, and I love it.
For our users looking to achieve even speedier deployment times, I offer up one reminder and one tip. First, analyze all of your script packages to ensure you are using the right means of customization. If you have some scripts that run for considerably longer than most other script packages, you may want to at least consider applying that customization by creating a custom image. You still need to adhere to the customization principles outlined here, but you may benefit from applying the customization in an image once and avoiding the penalty for applying it during every deployment. You may also be able to break this customization out with a combination of a custom image and script packages. For instance, instead of having a script that installs and configures monitoring agents, you may install the agents in a custom image and configure them during deployment. Being selective about how and when you apply customizations can go a long way in improving your deployment times.
In addition to the reminder above, I also have a tip. Take a look at all of the script packages you use in pattern deployments and look to see if there are any that you can apply in an asynchronous manner. In other words, identify customizations that need to start, but not necessarily complete as part of the deployment process. Going back to our example of configuring monitoring agents during the deployment process, it may be important to kick off the configuration script during deployment, but is it crucial to wait on the results? Maybe not. If it is not, consider defining the executable argument in your script package in a manner that kicks off the execution and proceeds -- i.e. nohup executable command &. This approach can save deployment time in certain situations.
My advice to users of WebSphere CloudBurst: keep pushing your deployment process! Pare as many minutes off the process as you can. I hope that the tips above help in that regard, and be sure to pass along other techniques that you have found helpful.
I was at a customer meeting the other day, and someone asked me if they could query WebSphere CloudBurst for an inventory of all of their virtual system deployments. This person was of course aware that he could go to the web console and very quickly view all of the virtual systems. What he wanted though was something that he could run to generate a report that contained all of this information. For a purpose like this, harnessing the WebSphere CloudBurst CLI is exactly the way to go.
I thought I'd write a simple CLI script that provides an example of how you could do this.
from datetime import datetime
outFile.write("WebSphere CloudBurst Virtual System Inventory\n")
outFile.write("Total virtual systems: " + str(len(cloudburst.virtualsystems)))
def writeVSDetails(outFile, virtualSystem):
outFile.write("\tVirtual system name: " + virtualSystem.name)
outFile.write("\tCreated from pattern: " + virtualSystem.pattern.name)
outFile.write("\tVirtual system status: " + virtualSystem.currentstatus_text)
created = datetime.fromtimestamp(virtualSystem.created)
outFile.write("\tVirtual system creation date: " + created.strftime("%B %d, %Y %H:%M:%S"))
outFile.write("\tTotal virtual machines: " + str(len(virtualSystem.virtualmachines)))
def writeVMDetails(outFile, virtualMachine):
outFile.write("\t\tVirtual machine name: " + virtualMachine.name)
outFile.write("\t\tVirtual machine display name: " + virtualMachine.displayname)
outFile.write("\t\tCreated from image: " + virtualMachine.virtualimage.name)
outFile.write("\t\tVirtual machine hypervisor: " + virtualMachine.hypervisor.name + " | " + virtualMachine.hypervisor.address)
outFile.write("\t\tVirtual machine IP address: " + virtualMachine.ip.ipaddress)
outFileLoc = sys.argv
outFile = open(outFileLoc, 'w')
for virtualSystem in cloudburst.virtualsystems:
for virtualMachine in virtualSystem.virtualmachines:
As a result of invoking this script using the CLI's batch mode, content is written to the file location supplied by the caller.
WebSphere CloudBurst Virtual System Inventory
Total virtual systems: 3
Virtual system name: Single server
Created from pattern: WebSphere single server
Virtual system status: Started
Virtual system creation date: January 15, 2010 16:37:20
Total virtual machines: 1
Virtual machine name: Standalone 0
Virtual machine display name: Single server cbvm-110 default
Created from image: WebSphere Application Server 188.8.131.52
Virtual machine hypervisor: Ruth ESX | https://<hypervisor_host>/sdk
Virtual machine IP address: <ip_address>
Virtual system name: Development WAS Cluster
Created from pattern: Custom WAS Cluster - Development
Virtual system status: Started
Virtual system creation date: January 18, 2010 14:08:46
Total virtual machines: 2
Virtual machine name: DMGR 0
Virtual machine display name: Development WAS Cluster cbvm-112 dmgr
Created from image: WebSphere Application Server 184.108.40.206
Virtual machine hypervisor: Ruth ESX | https://<hypervisor_host>/sdk
Virtual machine IP address: <ip_address>
Virtual machine name: Custom Node 1
Virtual machine display name: Development WAS Cluster cbvm-111 custom
Created from image: WebSphere Application Server 220.127.116.11
Virtual machine hypervisor: Ruth ESX | https://<hypervisor_host>/sdk
Virtual machine IP address: <ip_address>
Virtual system name: DB2 for development use
Created from pattern: DB2
Virtual system status: Started
Virtual system creation date: January 18, 2010 14:09:58
Total virtual machines: 1
Virtual machine name: DB2 Enterprise Server 32bit Trial 0
Virtual machine display name: DB2 for development use cbvm-113
Created from image: DB2 Enterprise 18.104.22.168 32-bit Trial
Virtual machine hypervisor: Ruth ESX | https://<hypervisor_host>/sdk
Virtual machine IP address: <ip_address>
I withheld IP addresses and host names above for obvious reasons, but if you ran the script against your environment you would see actual host name and IP address values. The script above is written once, and it can be subsequently run anytime you want an inventory of virtual systems running in your WebSphere CloudBurst cloud. There's other information available for virtual systems and virtual machines that I didn't show here, and you can retrieve it if necessary for your inventory report. In addition, I chose to print this information as regular text in a file supplied by the caller, but you might choose to generate the report in another format including XML, JSON, or anything else for that matter.
-- Dustin Amrhein
p.s. As with any sample code or script I provide here, the above is only a sample and offered as-is.
Jason McGee will be leading the second GWC Lab Chat this week on Wednesday, 4/20. The very timely topic is related to recent announcements from IBM regarding the IBM Workload Deployer (see previous posts). Entitled "Application-Centric Cloud Computing" the discussion will focus on the concept of deploying and managing your application workloads in a shared, self-managed environment rather than manually creating and managing the application middleware topologies. It places the focus on the application rather than the infrastructure. This concept promises to deliver greater simplicity, elasticity, and
density among other things. It can position your business to react more
quickly and efficiently to the increasing demands of your customers and
free you from the managing all of the details.
Many of you may have already heard Jason speak last week at IMPACT 2011 in the cloud mini main tent or perhaps at any number of other sessions that Jason was involved in. Jason is the key architect behind IBM's WebSphere cloud activities. Obviously, Jason understands the cloud space very well and has a clear view of the evolution into Application-Centric Cloud Computing. This GWC Lab Chat will provide the opportunity to get your questions answered and share your perspective on this technology.
Jason will provide a brief introduction to the concepts and ideas and then lead an open discussion. Put it on your calendar and plan to attend - and please plan to bring your questions and comments to help foster a rich discussion. We want to hear from you.
If you haven't registered yet it is not too late - learn more and register here. It is easy to register and there is no cost. This is a very timely event and a great way to dig a little more deeply into concepts you first heard at IMPACT or perhaps hear them for the first time. Don't miss it!
In the course of my job, I am lucky to be able to work with both enterprise users and business partners who are adopting and using the WebSphere CloudBurst Appliance. When it comes to the business partner camp, one of my absolute favorites is the Haddon Hill Group. The Haddon Hill Group is an IBM Premier Business Partner, and they have been an early adopter and vocal advocate of the WebSphere CloudBurst Appliance. They have extensive knowledge of the use of the appliance in enterprise accounts, and quite frankly, they are doing some really cool things with WebSphere CloudBurst.
Given the above, I was glad to see summarized results from their various implementations made available recently on the IBM site. The summary is fairly concise, so I encourage you to take a look at the results Haddon Hill Group is getting with WebSphere CloudBurst.
I am not going to rehash the contents of the results here, but there are a couple of things I want to call out. First off, Haddon Hill Group says that WebSphere CloudBurst can provide companies with a '100 times faster time to market' delivery experience. In a practical sense, this means reducing the amount of time to deliver WebSphere environments from 40-60 days on average to just hours. That is an eye-opening data point!
The other thing I want call out here is a quote from Phil Schaadt, President and CTO, Haddon Hill Group. I have had the pleasure of working with Phil and team, and I have heard him echo these same sentiments many times:
"The important thing about the IBM WebSphere CloudBurst Appliance is that it will dispense a WebSphere Application Server image onto your WebSphere Application Server environment or private cloud along with other products within the WebSphere stack, and that application server will be ready in a few minutes. You can do it in a clustered environment, and you can even roll out IBM WebSphere Process Server and get it right in a fully clustered environment with a database connection, in about 90 minutes. You can also easily manage all the configurations of IBM WebSphere Process Server that you need. All the steps that took up so much time and effort on the part of IT staff have been removed. The savings for companies with large WebSphere implementations can be in the millions."
It is always great to see clients putting our technology to use to produce tangible business value. Again, I encourage you to take a look at these reports. As always, I am eager to hear what you think, so leave me a comment or reach out to me on Twitter @damrhein.
When I talk with WebSphere CloudBurst users, the topic of custom virtual images comes up frequently. In some cases they simply want to customize a shipped IBM Hypervisor Edition, and in other cases they want to create a completely custom image. Creating a customized version of an IBM Hypervisor Edition is relatively easy since we give you extend & capture in WebSphere CloudBurst. Creating a completely custom image has historically been a bit tougher, mostly owing ot the fact that there was not a standard tool or process for image assembly. I am happy to say that today's publication of the IBM Image Construction and Composition Tool changes all that.
Watch a demo of the IBM Image Construction and Composition Tool
The primary purpose of the Image Construction and Composition Tool is to enable a modular approach to virtual image construction, while taking into account the typical division of responsibilities within an organization. The tool allows the right people within an organization to contribute their specialized knowledge as appropriate to the virtual image creation process. This means OS teams can handle the OS and software teams can handle the appropriate software. A separate image builder can then use both OS and software components to meet the needs of users within the organization. Best of all, the image builder does not need intimate knowledge of how to install or configure any of the components in the image. They simply need to know which OS and software components to use.
When using the Image Construction and Composition Tool, you start by defining the base operating system you wish to use for your images. You can do this by importing an existing virtual image with an OS already installed, providing an ISO for the OS, or pointing to a base OS image on the IBM Cloud. The bottom line is that you have necessary flexibility to start with your certified or ‘golden’ operating system build. Once you have the base OS image defined in the Image Construction and Composition Tool, you can start defining custom software for use in the images you will compose.
In the tool, bundles represent the software you wish to install within a virtual image. The definition of a bundle contains two major parts: Installation and Configuration. The installation component of a bundle tells the Image Construction and Composition Tool how to install your software into the virtual image. You provide a script or set of scripts that install the necessary components into your image, and you direct the tool to call these scripts. These tasks run once during the initial creation of the virtual image, thus allowing you to capture large binaries, long-running installation tasks, or other necessary actions directly into your image.
The configuration section of a bundle defines actions that configure the software installed into the image. Like with the installation tasks, you provide a script or set of scripts for configuration tasks. Unlike installation tasks that run exactly once, configuration scripts become part of the image’s activation framework and as such, run during each image deployment. Using the tool, you can define input parameters for configuration scripts and optionally expose them so that users can provide values for the parameters at image deploy-time. Configuration tasks are important in providing flexibility that allows users to leverage a single virtual image for a number of different deployment scenarios.
Once you have your base OS image and one or more bundles defined in the Image Construction and Composition Tool, you can compose a virtual image. To compose a virtual image, you extend the base OS image and add any number of bundles into the new image. A base OS image plus a set of bundles defines a unique image.
After you define the image you want to construct, you initiate a synchronize action in the Image Construction and Composition Tool. When you start the synchronize action, the tool first creates a virtual machine in either a VMware or IBM Cloud environment (based on how you configured the tool). Next, the installation tasks of each bundle you included in the virtual image run to install the required software. Finally, the tool copies the configuration scripts from each bundle into the virtual machine and adds them to the image’s activation framework. This ensures the automatic invocation of all configuration scripts during subsequent image deployments.
Once the image is in the synchronized state, you can capture it. Capturing the image results in the creation of a virtual image based on the state of the synchronized virtual machine. The tool also automates the generation of metadata that becomes part of the virtual image package. When the capture of the virtual image completes, you can export it from the Image Construction and Composition Tool and deploy it using WebSphere CloudBurst, Tivoli Provisioning Manager, or the IBM Cloud.
I am excited for users to get their hands on the Image Construction and Composition Tool. I believe it represents the first big step in helping users to design and construct more sustainable virtual images. Did I mention it is completely free to download and use? Visit the Image Construction and Composition Tool website for more details and a download link. I look forward to your comments and feedback.
IBM Impact 2011 was a wildly busy week! Customer meetings, entertaining keynotes, informative sessions, and hands-on labs packed the 6 days with more than enough action. I spent a lot of the week presenting sessions and conducting labs for the newly announced IBM Workload Deployer. As one would expect with any new announcement, we got tons of questions about IBM Workload Deployer. While I cannot capture all the questions and their answers here, I will try to cover some of the more prevalent ones below.
Question: What happened to WebSphere CloudBurst?
Answer: The short answer is, it simply went through a rename. WebSphere CloudBurst became IBM Workload Deployer v3.0. The version 3.0 acknowledges this is an evolution of what we started with WebSphere CloudBurst, which was at version 2.0. Why remove WebSphere from the name? The fact that this is now an IBM branded offering is more accurate as it is capable of deploying and managing more than just WebSphere software.
Question: What is new in IBM Workload Deployer?
Answer: While there are many new features that I will be talking about over the coming months, the most prominent new facet is the introduction of workload patterns (also referred to as virtual application patterns). As opposed to topology patterns (traditionally referred to as simply patterns in the WebSphere CloudBurst product), workload patterns raise the level of abstraction to the application level. Instead of focusing on application infrastructure and its configuration as you do with topology patterns, workload patterns allow you to focus on the application and its requirements. When using workload patterns, you provide the application, attach policies that specify functional and non-functional requirements, and deploy. IBM Workload Deployer handles deploying and integration the middleware infrastructure necessary to support the application, and it automatically deploys your application on top of that middleware. In addition, IBM Workload Deployer manages the application runtime in accordance with the policies that you specify in order to provide capabilities such as runtime elasticity.
Question: If I am a current WebSphere CloudBurst user, what does this mean for me?
Answer: Not to worry. You will be able to use all of your WebSphere CloudBurst assets (patterns, scripts, images) in the new IBM Workload Deployer. All of the capabilities previously in WebSphere CloudBurst are present in IBM Workload Deployer (terminology may vary slightly -- topology pattern instead of just pattern for instance). Additionally, we continue to expand on the functionality that you are familiar with from WebSphere CloudBurst. This includes updates for Environment Profiles, new IBM Hypervisor Edition images, new pattern building capabilities, and more. Stay tuned for more information about these new features and for information on how you can move your WebSphere CloudBurst resources to the new IBM Workload Deployer.
Question: How do I choose between using workload and topology patterns?
Answer: There are a number of factors that will lead you to using either workload patterns, topology patterns, or both. The primary decision point will be how much control you really need (not want). When using workload patterns, you sacrifice some customization control over the configuration, integration, and administration of the middleware application environment since the workload pattern and management model abstracts away the 'guts' of the system. Everything about the workload pattern is application-centric. On the other hand, topology patterns give you intimate control over the configuration, integration, and administration of the middleware application environment. As a general rule of thumb, if your application requirements match the capabilities of a workload pattern, that is the way to go as it can greatly reduce complexity and cost associated with deployment and management. If a workload pattern does not meet the needs of your application, topology patterns can still greatly reduce cost and complexity and you can tailor them to fit almost any need. Beyond generalities, there is no hard and fast rule for choosing one over the other. It comes down to understanding your application environment and its needs.
Question: Is IBM Workload Deployer an appliance like WebSphere CloudBurst?
Answer: Yes, it is still an appliance, but an updated one! The new appliance is 2U, and it provides more storage, processing power, and memory. It is still just as easy to setup, but just slightly bigger.
Well, that is all for now, but I will be back many times over the coming months with more information. In the meantime, if you have any questions, please leave them in a comment below.
During the week of IMPACT this year, we announced the launch of the WebSphere CloudBurst Samples Gallery. You can go to this gallery to find and download sample script packages, CLI scripts, and other tools that we hope help you in your endeavors with the appliance. The samples are free to use and offered in an "as-is" fashion.
While I certainly will not write about each and every sample we post out there, I did want to bring your awareness to a new one I just put up today. The new sample is neither a CLI script nor a script package, though you will find it in the script packages section of the gallery. Instead, the new sample is a tool that you can run to produce WebSphere CloudBurst script packages.
Specifically, the tool runs against a target WebSphere cell to produce a WebSphere CloudBurst script package that encapsulates that cell's configuration. The tool works by running the backupConfig command against the target cell. It packages the ZIP file that results from running the command into a special WebSphere CloudBurst script package that you can include in patterns which match the source cell in node quantity and type.
The script package produced by the tool packages logic to run the restoreConfig command using the backed up configuration from the source cell. This will apply the source configuration to a new WebSphere Application Server cell created as the result of deploying a pattern. In addition, the script package contains logic to handle the possibility of changing cell, node, and host names in the target environment.
The tool’s purpose is to help you accelerate the process of importing your existing WebSphere Application Server environments into the appliance as patterns (which is a problem I believe many of you would like to solve). It certainly does not handle everything you need to do to import environments. In fact, it has the same limitations as the backupConfig/restoreConfig utilities in WebSphere Application Server. However, I do believe that it makes it a little easier to start moving your existing environments into the appliance as new WebSphere CloudBurst patterns.
Check out this video to see a quick overview of the tool, and then go download it for free from the samples gallery. The ZIP file that you download has a readme file that gives specific detail about how to use this sample tool. As always, please let me know if you have any questions or feedback.
Virtual image parts play a huge role in WebSphere CloudBurst. When crafting your own customized patterns, you include anywhere from 1 to n parts from as many different virtual images as is necessary. These parts represent the different node types or personalities within a given Hypervisor Edition image, and form the basis of your pattern. When you deploy a pattern, such as the one pictured below, WebSphere CloudBurst creates a distinct virtual machine for each part.
This means that after deploying the above WebSphere Application Server pattern, you will have four virtual machines comprising your virtual system. This gives you a clean separation of concern by providing a unique container for each of your application environment nodes. This can attribute to performance optimization, increased availability, and much more. However, this approach is not suitable to all use cases. In some scenarios, especially when trying to control costs and increase consolidation, you may want to deploy a multi-node WebSphere Application Server environment within a single virtual machine. Based on what I showed you above, you might think our approach in WebSphere CloudBurst makes this impossible, but you would be overlooking an important component of patterns.
That component is of course the second building block of patterns... script packages. As you probably know, script packages allow you to supply just about any customization you want. In the case that you want a single virtual machine to host a number of WebSphere Application Server nodes, maybe even an entire cell, all you need to do is supply a script package that constructs the necessary nodes during deployment. In fact, you don't even have to write the script package. You can use the free sample in our samples gallery. As seen in the pattern below, you include this script package on a sole deployment manager part in a pattern.
The script script package provides parameters that define the node name, number of custom nodes, and number of web server nodes you want in your cell. During the deployment process, the script takes this information and constructs the cell you define. This includes creating the custom and web servers nodes and federating the custom nodes, thus completing the creation of your WebSphere Application Server cell. In this case, the script package provides deployment flexibility that is sometimes a necessity, and it is just another example of the many degrees of flexibility enabled by the script package design.
I should point out that a part in a pattern does not always map to a single node. For instance, in the case of WebSphere Process Server, there is a part that represents a complete, multi-node golden topology encapsulated within a single virtual machine. However, if you find yourself using images that do not contain these multi-node parts, rest easy knowing script packages provide you the flexibility you need.
I’m going to take a different approach this week in the blog. Instead of me telling you about some of the features or uses of WebSphere CloudBurst, I thought I would catch up with someone using the product everyday, WebSphere Test Architect Robbie Minshall. Robbie is responsible for a team of testers that harness a lab of over 2,000 physical machines to put our WebSphere Application Server product through some pretty rigorous testing. Toward the beginning of this year Robbie’s team started to leverage the WebSphere CloudBurst Appliance in order to create the WebSphere Application Server environments needed for their testing.
Robbie, can you tell us a little bit about what the WebSphere Application Server test efforts entail?
In WebSphere Application Server development and test we have two primary scenarios. The first is making sure that developers have rapid access to code, test cases and server topologies so that they can write code, test cases and then execute test scenarios on meaningful topologies. The second scenario is an automated daily regression where in response to a build, we provision a massive amount of WebSphere Application Server topologies and execute our automated regression tests.
Previously we have supported these scenarios through the deployment of the Tivoli Provisioning Manager for operating system provisioning, some applications for checking out environments, and then a lot of automation scripts for the silent install and configuration of WebSphere Application Server cells.
Given those scenarios and the existing solution, what are your motivations for setting up a private cloud using WebSphere CloudBurst Appliance?
We are supporting these scenarios through a pretty complicated combination of technologies. These include silent WAS install scripts, wsadmin configuration scripts, a custom hardware leasing application and the utilization of Tivoli Provisioning Manager for OS Provisioning. This solution is working very well for us though as always we are looking for areas to improve, opportunities to simplify and to reduce our dependency on investment in our custom automation scripts. Mainly, there were 3 areas where we wanted to improve our framework: Availability, Utilization and Management. This is why we started looking to the WebSphere CloudBurst Appliance.
Can you expand a bit on what you are looking for in those three areas?
The first focus area we have is availability of environments. We really wanted to lower the entry requirement for the skills and education necessary to get a development or test environment. Setting up these environments has just been too hard, too time consuming, and too error prone. Using WebSphere CloudBurst we can provide an easy push button solution for developers to get on-demand access to the topologies they need.
The second area we are looking for significant improvements on is hardware utilization. Our budgets are tight and in our native automation pools we are only using between 6-12% of the available physical resources. In order to improve this we were looking at leveraging virtualization. WebSphere CloudBurst offers the classic benefit of virtualization with the nice additions of optimized WebSphere Application Server placement and really good topology and pattern management. In our initial experiments we were able to push the hardware utilization up to 90% of physical capacity and consistently were leveraging around 70% of our physical capacity.
Finally we are looking to improve and simplify our management of physical resources and automation. We work in a lot of small agile teams and organizational priorities change from iteration to iteration. Not only does WebSphere CloudBurst allow us to maintain a catalog of topologies or patterns for releases but it also allows us to adjust physical resource allocation to teams through the use of sub clouds or cloud groups.
Basically we felt that WebSphere CloudBurst would improve the availability of application environments, enhance automation, and improve hardware utilization all with very low physical and administrative costs.
What were some of the challenges involved with getting a cloud up and running in your test department?
One of our challenges seems like it would be common to many scenarios, especially in today’s world. Our budget for new hardware to build out our cloud infrastructure was initially very limited. Most cloud infrastructure designs depict very ideal hardware scenarios including SANs, large multicore machines, and private and public networks within a dedicated lab. Quite frankly we did not have the budget to create this from scratch. It was important for us to demonstrate value and data to warrant future investment in dedicated infrastructure. After some performance comparisons we were very happily surprised to see that we could leverage our existing mixed hardware within a distributed cloud. The performance of application environments dispensed by WebSphere CloudBurst on many small existing boxes in comparison to large multicore machines with a SAN was very comparable. This allows us to leverage existing hardware, with minimal investment all the while demonstrating the value and efficiencies of cloud computing. That data in turn has allowed us to obtain new dedicated hardware to iteratively build up a larger lab specifically for use with WebSphere CloudBurst.
Specifically with WebSphere CloudBurst, are there any tips/hints you would offer users getting started with the appliance?
Sure. First, we quickly realized as we added hypervisors to our WebSphere CloudBurst setup it was critical to have someone with network knowledge on hand. This is because the hypervisors came from various sections of our lab, and we really needed people with knowledge of how the network operated in those different sections. Once we had the right people we were able to setup WebSphere CloudBurst and deploy patterns within an hour and a half.
Moving forward we continued to have challenges as we dynamically moved systems between our native hardware pool and our cloud. Occasionally the WebSphere CloudBurst administrator would move a system into the cloud but incorrectly configure the network or storage information. This lead to some misconfigured hypervisors polluting our cloud. We overcame this, quite simply and satisfactorily I may add, by creating some simple WebSphere CloudBurst CLI scripts which add the hypervisors, test them individually, by carrying out a small deployment to that hypervisor, and then move the correctly configured hypervisors into the cloud after verifying success. Misconfigured hypervisors go into a pool for problem determination. This has allowed us to maintain a clean cloud, and we are able to dynamically move our hardware in and out of the cloud to meet our business objectives.
We also use the WebSphere CloudBurst CLI to prime the cloud so to speak. Before using a given hypervisor in our cloud, we execute scripts that ensure each unique virtual image in our catalog has been deployed to each of our hypervisors at least once. When the image is first deployed to a hypervisor, a cache is created on the hypervisor side of the connection, thus meaning subsequent deployments do not require the entire image to be transferred over the wire. This gives us consistent and fast deployment times once we are using a hypervisor in our cloud.
I would assume that like many applications deployed on WebSphere Application Server, your team’s applications have several external dependencies. Some of these dependencies won’t necessarily be in the cloud, so how did you handle this?
You’re right about the external dependencies. Our applications and test cases run on the WebSphere Application Server but are dependent upon many external resources such as databases, LDAP servers, external web services etc. WebSphere CloudBurst allows us to deploy WAS topologies in a very dynamic and configurable way but the 1.0.1 version does not allow us to deploy these external resources in the same manner. This was overcome by using script packages in our patterns. These script packages allow us to associate our test applications with various patterns we have defined. The script package definition also allows us to pass in parameters to the execution of our scripts. We supply these parameter values during deploy time, and these values are used to convey the name or location of various external resources. The scripts that install our applications can access these values and ensure the application is properly integrated with the set of resources not managed by the appliance.
What is your team looking to do next with WebSphere CloudBurst and their private cloud?
The next challenge on our plate is to keep up with the demand of our expanding cloud and to develop a more dynamic relationship between our native pools and our cloud using the Tivoli Provisioning Manager. These are fun challenges to have and we look forward to sharing our progress.
I'm glad I got to spend some time with Robbie to glean some insight into their work and progress with WebSphere CloudBurst. I hope this information was useful to you. It's always nice to hear about a product from practitioners who can give you hints, tips, gotchas, and other useful information. Be sure to let me know if you have any questions about what Robbie and his team are doing with WebSphere CloudBurst.
If you are reading this blog then I am pretty sure that you are interested in the agility that can be achieved by rapidly provisioning middleware systems and standing up virtual applications in a private cloud environment. However there are other aspects of agility that you should also consider. One such aspect is the ability to build applications that can be easily maintained, updated, and extended. This is where OSGi technology comes into the picture.
If you have been working with the IBM Workload Deployer (or watching some IBM Workload Deployer demos) you may have noticed a category of components in the virtual application builder called OSGi Components.
Maybe you already know all about OSGi applications and the value they bring to an enterprise. Or, perhaps you noticed this and decided that you would search for some more information on this odd acronym and just what an OSGi application is all about.
In a nutshell OSGi technology is a way to define dynamic modules for Java. It provides a standard way to encapsulate components (called bundles) with metadata that define versioned package dependencies, service dependencies, packages exported, services exported, etc... basically everything you need to know about this bundle so that it can be connected up with other bundles to support a particular solution. These bundles can then be grouped together into applications and dynamically wired to fulfill necessary dependencies at runtime. The OSGi framework provides all of the necessary capability to manage the dependencies and resolve any problems.
Those who leverage OSGi technology benefit from improved time-to-market and reduced development costs. The loose coupling provided by the OSGi framework reduces maintenance costs and facilitates the dynamic delivery of components in a running system. Of course there's a lot more to it than just that ... involving portability across different environments, achieving the appropriate level of isolation or sharing within an environment, and integrating with the many different technologies and patterns already available today. I don't think I know enough about OSGi to do it justice here. But fortunately for me (and you) there are several experts who can make it all clear.
One such expert is Graham Charters and there is a great opportunity to hear him introduce this topic and also participate in a dialogue about the concepts and what they mean for your business. Graham will be leading a Global WebSphere Community Lab Chat on Wednesday of this week (July 20th) entitled: How can OSGi make your enterprise more agile. Graham is the IBM technical lead in the OSGi Alliance Enterprise Expert Group and an active participant in the open source community implementing many of these standards. So register now for this free session and learn how OSGi can make your enterprise even more agile.
Starting in WebSphere CloudBurst 2.0, there are different levels of elasticity that you can achieve in your WebSphere deployments. As I mentioned in a previous post, the Intelligent Management Pack allows you to define dynamic clusters. This means cluster membership and the number of instances of a given application adjusts on the fly to meet SLAs for your application. This enables a more dynamic environment as opposed to static cluster definitions, but there is a layer of elasticity below this that bears exploring.
Dynamic clusters work with WebSphere nodes that already exists. Users define the nodes available for use by a dynamic cluster, and the runtime uses SLAs and current system state to determine the actual nodes used and application instances started. So, what if you need more nodes than what you currently have in a given WebSphere environment? A dynamic cluster will not create a new node, so you have to define extra nodes. Starting in WebSphere CloudBurst 2.0, this is as easy as pushing a button.
Dynamic virtual machine operations allow you to add and remove nodes on the fly for a given virtual system. For instance, take the pattern in the picture below:
If you were to deploy this pattern, you would end up with a WebSphere Application Server cell with a node makeup similar to the below:
Now that the environment is out there (in mere minutes I should mention), suppose you want to add more nodes? Before WebSphere CloudBurst 2.0, you could have done it, but it would involve creating another pattern with a custom node part and deploying it. This results in two different virtual systems and complicates the maintenance stream. Now, in WebSphere CloudBurst 2.0, you can simply click a button to add a node to the existing virtual system.
From the virtual systems view, if you expand the virtual machines, by each virtual machine you will see an Actions column with a View link. If you want to add a node to the environment shown above, you simply click the View link, and then click the clone icon highlighted in green below:
WebSphere CloudBurst prompts you for the number of nodes to add. You make the selection and then click OK. The appliance creates the new node and federates it into the cell for you. For instance, if you chose to add a single node, at the end of the clone not only would you have another virtual machine in your virtual system, but also another node automatically federated into your WebSphere Application Server cell:
On the flip side, you can remove a node by clicking the delete icon in the same dialog as the clone icon above. This removes the node from the WebSphere Application Server cell and deletes the virtual machine.
The ability to easily add and remove virtual machines from your WebSphere CloudBurst virtual systems enables a very valuable level of elasticity. Now you can very easily add and remove nodes on the fly based on the current demands of your system. As always, let me know if you have any questions or comments.
One of the most powerful features of WebSphere CloudBurst is the ability to take one of the WebSphere Application Server Hypervisor Edition virtual images that are shipped with the appliance and extend it to a produce a custom virtual image. This allows users to begin creating customized environments from the bottom up, starting with the operating system. To put it in better context, let's take a look at a couple of scenarios where this feature comes in quite handy.
First off, a very common need for our customers is the ability to continually monitor their application environments. For instance, you may install Tivoli monitoring agents on all of your machines hosting WebSphere Application Server processes and configure those agents to report back to a management server. This is a great case for image extension in WebSphere CloudBurst.
In this scenario, you would start by extending an existing WebSphere Application Server Hypervisor Edition image. WebSphere CloudBurst creates a running virtual machine based off of the selected image, and you log into that virtual machine and install the Tivoli monitoring agents. Once the installation is done, you capture the virtual image back into the WebSphere CloudBurst catalog and use the new image to build a custom pattern. The last step is to include a script package on this custom pattern that, upon deployment, will configure the installed monitoring agents to report back to your desired management server.
Another use case is likely to be of interest to you if you are using WebSphere Virtual Enterprise (or something similar), and you could benefit from the same ease of provisioning for those environments that WebSphere CloudBurst brings to WebSphere Application Server environments. You can use the same customization combination above (image extension and custom scripts) to enable WebSphere CloudBurst to essentially dispense WebSphere Virtual Enterprise cells.
Again, this scenario starts off by extending a WebSphere Application Server Hypervisor Edition virtual image. Once the virtual machine for the extension is created by WebSphere CloudBurst, you log in and install the WebSphere Virtual Enterprise product. After the installation is done, you capture the image and store it in the catalog. Next, you build a custom pattern based off of this image and include script packages that, upon deployment, augment the various parts in the pattern from WebSphere Application Server profiles to WebSphere Virtual Enterprise profiles. (You may wonder why you wouldn't just create the WebSphere Virtual Enterprise profiles during the image extension process. This is because during image extension, you cannot make changes to the virtual disk that contains the WebSphere Application Server profiles. Any changes made to the profiles will be wiped out during the capture process.)
There are countless more scenarios for creating custom virtual images in WebSphere CloudBurst. To name a few, you may want to install JDBC drivers that are common to almost all of your application environments, install required anti-virus software, or just make operating system configuration changes. All of these things can be accomplished through the image extension and capture process. Look for an article coming out soon that will discuss and explain, in much greater detail than I provided here, the process of installing and configuring Tivoli monitoring agents in environments dispensed by WebSphere CloudBurst. In the meantime, if you have any questions or comments, drop us a line here or check out our forum.
As I have mentioned before, IBM Workload Deployer v3.0 introduces choices in pattern-based deployment models. One of those models, virtual system patterns, is a carry over from the WebSphere CloudBurst Appliance. When you use virtual system patterns in IBM Workload Deployer, you can take advantage of all of the techniques you put to use in WebSphere CloudBurst. This is certainly good news for current WebSphere CloudBurst users, but it goes a bit further. Instead of simply maintaining the status quo with virtual system patterns, which would have been reasonable considering the introduction of virtual application patterns, we chose to continue to expand on your customization options for this pattern deployment model. In particular, I want to discuss three new features in IBM Workload Deployer that may help you to better construct and manage virtual system patterns.
The first new feature is one that I have been eagerly awaiting. In the new version of the appliance, we provide you with the ability to specify part and script package ordering in your pattern. This means that, within the virtual system pattern editor, you can tell IBM Workload Deployer in which order to start the virtual machines in your pattern, and you can specify in which order to invoke the script packages within the pattern during deployment. This eliminates the need for special script invocation orchestration logic in your pattern (I had customers resorting to a semaphore like approach using a shared file system), and it allows you to be more declarative about the virtual machine bring-up process. There are constraints, specifically with the part ordering. Some images will impose an implied part start-up order that you cannot change. For instance, deployment manager parts in the WebSphere Application Server Hypervisor Edition image must start before custom node parts. The good news is the pattern editor will not allow you to specify a part start-up order that violates these constraints. The image below shows an example of the ordering view in the virtual system pattern editor.
Another new feature that may influence the way you build virtual system patterns is the introduction of Add-Ons. You can think of Add-Ons as special script packages that you can include in your virtual system pattern that perform system-level configuration actions. Specifically, you can include add-ons in your virtual system pattern to add an operating system user, add a virtual disk, or add a NIC during the deployment process. You include Add-Ons in your pattern by simply dragging and dropping them onto a part in your pattern, just as you do with script packages today. The difference between script packages and Add-Ons is that IBM Workload Deployer will ensure the invocation of all Add-Ons before any other scripts run during deployment. We include default Add-On implementations for adding a user, disk, and NIC.
The last new feature I want to talk about today has more to do with how you manage or govern the deployment of virtual system patterns. In WebSphere CloudBurst 2.0, we introduced the idea of Environment Profiles as a way to extend your customization reach into the deployment process. Initially, these profiles gave you the ability to directly assign IP addresses to virtual machines in your deployment, declaratively specify virtual machine naming formats, and easily split a single pattern deployment across multiple cloud groups. In IBM Workload Deployer, you will be able to use these same profiles to set resource consumption limits for pattern deployments. In particular, you will be able to set cumulative limits for virtual CPU, memory, storage, and software licenses used by deployments tied to a specific profile, thereby giving you finer-grained control over cloud resource consumption. The picture below shows the new resource limit aspects of environment profiles.
Virtual system patterns are key in the deployment model choices for IBM Workload Deployer. Not only did we carry the concept over from WebSphere CloudBurst to IBM Workload Deployer, but we made it even better. Expect this trend to continue!