Using WebSphere DataPower in IBM PureApplication System

Content

Introduction

First, a summary of the differences between the DataPower Virtual Edition images compared to other product images in PureApplication System is discussed in this article. Then, you will learn how to use the PureApplication System web console to import the DataPower XI52 Virtual Edition into the image catalog, create a virtual system pattern using the XI52 part, and deploy the pattern into the cloud. Finally, an example script package is provided that allows you to perform advanced configuration and install features into the DataPower appliance on deployment.

DataPower background

The DataPower product portfolio offers two types of virtual appliances:

• IBM WebSphere DataPower Integration Appliance XI52 Virtual Edition
• IBM WebSphere DataPower Service Gateway XG45 Virtual Edition

The virtual appliances offer the same functional features that you are familiar with in the DataPower physical appliances, without some of the physical security features such as a Trusted Platform Module (TPM), crypto acceleration card, Hardware Security Module (HSM) card, and tamper-proof casing.

Each virtual edition product is offered through a non-production or a production license with different pricing. The non-production license is designed for development purposes and has all of the licensed add-on features enabled, except for the Tibco optional feature. The production license is designed for enterprise grade production deployments and allows you to install the add-on licensed features using the same method as physical appliances. Configuration is transferable between virtual and physical appliances, so you can export configuration from a virtual appliance and import it into physical appliances, or vice versa. This makes the non-production virtual edition a good option for development with the intent of using a physical appliance in production.

This article uses BM WebSphere DataPower Integration Appliance XI52 Virtual Edition. This is a purpose-built, virtual appliance that delivers rapid data transformations and security enforcement for cloud and mobile applications. It provides security-enhanced and scalable business integration in an edge-of-network security gateway.

Here are some of the XI52 Virtual Edition features:

• Provides streamlining and security for service-oriented architecture (SOA) and enables any-to-any data transformation, including wire-speed XML, JSON, text, binary, and custom formats.
• Helps reduce integration costs with protocol mediation, protocol bridging, transport mediation, multistage pipeline processing, and content-based message routing.
• Optimizes bridging between wire-line messaging protocols, including IBM WebSphere MQ, IBM WebSphere JMS, FTP, and HTTP.
• Provides centralized service governance and service-level agreement (SLA) management by integrating with IBM WebSphere Service Registry and Repository and IBM Tivoli Composite Application Manager.
• Supports direct-to-database connections for IBM DB2, Oracle, and Sybase, and supports standards and connectivity including data validation, field-level security, web services management, and application optimization.

For more information about WebSphere DataPower appliance, see the WebSphere DataPower SOA Appliances page.

PureApplication System background

IBM PureAplication System is an integrated system that combines hardware resources (compute nodes, storage, and networking) with software resources, using virtualization to maximize the utilization of the system. Workloads (such as transactional applications, web applications, or data-centric applications) are supported via the use of "patterns", which offer abstract views of the software components of a solution, for example, web applications or databases. PureApplication System also supports defining topologies consisting of virtual images and scripts that are used to configure the resulting virtual machines. This includes installation of middleware, or the integration of multiple nodes (for example, virtual machines) with each other. These topologies are defined as so-called "virtual system patterns" and can be deployed repeatedly and predictably, because every step of the deployment process is codified.

Workloads that are deployed as part of a pattern, and which run in PureApplication System, are easily monitored and managed through the system. It offers a single point of control for all of the virtual machines that are running with multiple levels of monitoring and logging, as well as emergency fix support.

The ability to quickly deploy environments in a predictable and repeatable fashion, literally at the push of a button, and to do so in a way that delivers highly available and manageable topologies, is a very attractive proposition for the use of PureApplication System. One type of workload that we expect to become a common use case within the collection of patterns available for PureApplication System is that of an Enterprise Service Bus or, more generally, an Integration Layer, which connects multiple frontend consumers to multiple back-end providers of services. Appliances like the IBM WebSphere DataPower appliance are central to many integration layer implementations, covering, for example, the security of the interactions between consumers and providers. The remainder of this article will describe how you can leverage the WebSphere DataPower appliance within a workload pattern deployed on PureApplication System.

A secure appliance

Originally, DataPower products were only offered as secure rack-mounted appliances utilizing special purpose hardware and a custom operating system. DataPower appliances are widely deployed in enterprise demilitarized zones (DMZs), which have driven requirements for critical security features in the DataPower product offerings. Some of the physical security features are not possible when running as a guest on a standard hypervisor, but the virtual offering has retained most of the secure design principles that are available on the physical appliances. To retain some of the security features present in the physical appliances, the DataPower virtual image was built using a custom build process instead of using the IBM Image Composition and Construction Tool (ICCT).

Some of the differences between the DataPower virtual image and other products in the PureApplication System catalog are:

• DataPower is built as an appliance that includes a custom, special-purpose, secure operating system. It is not built by deploying an application onto a general-purpose operating system.
• The operating system is installed on an encrypted virtual disk.
• General-purpose access to the file system is not allowed. The only method of installing files onto the appliance is through an upgrade method that uses a signed and encrypted firmware image.
• There is a custom command line interface (CLI) with a specialized command set instead of a standard shell like BASH.

Due to the custom nature of the DataPower appliance, the following PureApplication functions are not supported at this time:

• Add-on scripts: There is no standard shell access.
• Add-on disks: DataPower is designed to manage exactly two disks.
• Add-on NICs: DataPower virtual firmware only supports between 1 and 4 network interface cards (NIC).
• Image extensions: General purpose file system access is not allowed.

Requirements

The pattern described in this article requires the following:

• IBM PureApplication System W1500 Version 1.1
• IBM WebSphere DataPower XI52 virtual appliance Version 6.0.0.0 or later (6.0.0.1 is used in the examples)
• 9 virtual CPUs available:
  • 2 x 4 CPUs for two DataPower nodes
  • 1 CPU for Core OS node
• 18 GB memory available:
  • 2 x 8 GB for two DataPower nodes
  • 2 GB for Core OS node

Importing the image

1. To get started in PureApplication System with DataPower, you need to import the OVA file into the image catalog. Before you can import the OVA, you need to make it available on an HTTP server, such as an IBM HTTP Server. The examples in this article use DataPower XI52 Virtual Edition Version 6.0.0.1.
2. To import the image, log in to the PureApplication console, switch the view to the Workload Console, click on Catalog > Virtual Images, click on the + icon, and provide the URL of the OVA that you want to import. If the image is being hosted by a server that requires authentication, then you also need to provide the user name and password to authenticate. The time required for downloading the OVA ranges from a few minutes to an hour, depending on the bandwidth and latency of the network between PureApplication System and the HTTP server that is hosting the OVA.
3. After the OVA is imported into the image catalog, you can see that the image includes a single part that you can use in virtual system patterns. You need to accept the end user license agreement before you can use this part. To accept the license, locate the image in the catalog and click Accept License in the image details panel. Read the license carefully before clicking the OK button to accept it. After accepting the license, you can see the image in the image catalog. It is ready to be used as a part in virtual system patterns as shown in Figure 1.

   Figure 1. The DataPower production image in the catalog

Deploying a simple pattern

There are three types of patterns in PureApplication System: virtual application patterns, virtual system patterns, and database patterns. You can only use the DataPower part in virtual system patterns.

1. You can create a new virtual system pattern by clicking Patterns > Virtual Systems and then clicking the + icon. In the pop-up dialog, provide a name and description for your pattern and click OK.
2. After clicking OK to create a pattern, you see the details of the pattern. In the menu bar on the right side of the browser, click the Edit button to launch the pattern editor to build your pattern.
3. The pattern editor provides a palette for you to wire parts into your pattern. On the left side of the palette, find the DataPower part by scrolling through the list of parts or typing DataPower into the search box. Select the DataPower part by clicking it and then drag it onto the palette on the right side of the browser. This adds the DataPower part to your pattern. As shown in Figure 2, the DataPower part has two public NICs, in addition to the two base NICs provided for all images. This gives you four total NICs: one private NIC for PureApplication management and three public NICs for use in traditional DataPower management and data plane traffic.

   Figure 2. The DataPower production part on the Pattern Editor palette

4. Click Done Editing in the top-right section of the browser to stop editing the pattern.
5. To deploy the pattern, click the Deploy icon in the top-right section of the browser.
6. Provide a name for the instance, choose the correct environment profile, and click the Configure virtual parts link to expand the parts section.
7. Click the WebSphere DataPower XI52 Virtual Edition link to configure the DataPower part.
8. The deployment properties for the DataPower part allow you to configure the same items that are configured on the first boot interview when the appliance is started outside of PureApplication System. All of the properties have sensible defaults except for the two add-on network interfaces and the admin password. You must select an IP group for each of the two add-on NICs and you must provide an admin password that is at least 6 characters long. After providing these parameters, click OK on the Properties dialog.
9. After the values for the DataPower part properties have been provided, you see a green check mark next to the part and the parts section. Click OK to deploy the pattern into the cloud.
10. The pattern takes a few minutes to deploy. If it is successful, you see a green arrow icon next to it. You can test the deployment by accessing the DataPower web GUI. The URL for the DataPower web GUI is https://<ip address>:<port>. You can find the IP address of the deployed instance by expanding the details section of the Instance view. The port is "9090" unless you changed it to a non-default value during deployment.

Adding a script package to upgrade and configure

Virtual system patterns typically contain a virtual machine hosting Linux-, Windows-, or AIX-based images, on top of which the middleware or other software is installed. Additional configuration is then done using script packages, which can bundle additional files that are loaded into the virtual machine and then executed.

The virtual image hosting the DataPower appliance is different. It is not based on a general purpose operating system, and thus cannot host additional scripts. In other words, you cannot inject script packages into it using the virtual system pattern editor. All additional configuration after the virtual appliance has been deployed and started has to happen externally.

The "DataPower Control VM" pattern

The approach we recommend for hosting script packages that can be executed against virtual DataPower appliances is to deploy a regular Linux VM, with all the appropriate script packages included and set to execute "when I initiate it…" This "DataPower control" VM can either coexist in the same pattern with the appliances, or it can be in its own pattern and be deployed separately. It can then be used to control any virtual appliance deployed in the system.

Figure 3. The DataPower Control VM pattern

Figure 3 shows an example for such a control VM. It includes several script packages, all of which are executed on demand, and which address specific administrative functions that you can execute on the targeted virtual appliance.

The interface we use for the remote administration is the DataPower XML Management Interface. This interface uses SOAP over HTTP to access the appliance. The default port for the XML Management interface is 5550, even though you can set it to a different value during the deployment of the virtual appliance. In order to connect to this port, you have to update the firewall rules on the control VM to allow outgoing traffic. The "DataPower – Open firewall port" script handles this. It contains the following line:

  /sbin/iptables -I OUTPUT 6 -p tcp --dport $DP_ADMIN_PORT -j ACCEPT

The $DP_ADMIN_PORT variable is defined on execution of the script and is set to 5550 by default. This script must be run before any other DataPower configuration can occur, which is why its execution mode is set to "at virtual system creation and when I initiate it".

Configuration script packages for DataPower

As mentioned above, the configuration approach we use is based on DataPower's XML Management interface. This means our scripts all must build a SOAP envelope that is appropriate for the request we want to send. In this article, we will show you a sample collection of different script packages, all of which follow the same basic structure. This makes it easier for you to create scripts for your own purposes.

Let's take a look at the "DataPower – Create Domain" script package (this as well as other scripts and patterns introduced in this article are found in the Download section of this article). It requires a set of parameters to be set on its execution, as shown in Figure 4.

Figure 4. Parameters for the "Create Domain" script package

The first parameter asks for the IP address of the DataPower appliance that the request will be sent to. In the context of this article, we expect this to be a virtual appliance deployed in a PureApplication System, but it can actually be any physical DataPower appliance, too, as long as its IP address and port are reachable.

To find the IP address of the target appliance, simply open the Virtual System Instance view in the PureApplication System console, find the pattern instance with the appliance you want to configure, and expand the Virtual Machines section to find the network interface, as shown in Figure 5.

Figure 5. The IP address of the Virtual DataPower appliance

The second parameter is the port number, and 5550 is the default. The next parameters ask for the user ID and password to be used to authenticate with the appliance. Finally, the last parameter lets you enter the name of the new domain that is created.

Note that all of the script packages we use for this article have the same set of parameters at the beginning, namely IP address of the target appliance, as well as port number, user ID, and password.

The script itself is a shell script that simply calls a generic Python program named "dpManage.py". This Python program uses a generic argument parser to read the passed arguments and delegates all request specific processing to another Python program identified by the "request" parameter. For example, the "Create Domain" script package contains a "createDomain.py" file that reads in the "NEW_DOMAIN" parameter and sets up the basic SOAP message that is sent to the appliance, as shown in Listing 1.

   from string import Template  def add_arguments(parser):   parser.add_argument('--newdomain',required=True)    def get_request(args):   template = Template('''<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">   <soapenv:Header/>     <soapenv:Body>       <dp:request xmlns:dp="http://www.datapower.com/schemas/management">         <dp:set-config>           <Domain name="${NEWDOMAIN}">             <NeighborDomain class="Domain">default</NeighborDomain>           </Domain>         </dp:set-config>       </dp:request>     </soapenv:Body>   </soapenv:Envelope>''')   return template.substitute(NEWDOMAIN=args.newdomain)

You can create your own script package for any required admin function following the same approach as shown here, namely by using the dpManage.py program augmented by a request specific module for specific parameters and setting up the SOAP request message. Creating the connection to the target appliance as well as sending the request message is handled by dpManage.py for all requests.

Automated configuration using the PureApplication CLI

Besides executing these configuration scripts via the PureApplication System console, you can also invoke the script packages in an automated way using the Command Line Interface or CLI. You can use either the CLI can in an interactive mode, or by passing in a Python program that executes the desired function. Do not confuse this CLI with DataPower's own CLI. In this case, you issue commands to PureApplication System to launch the script package defined above. You are not interacting with DataPower's CLI.

Listing 2 shows an example Python program that executes the "Create Domain" script package via the CLI. Note that the script package parameters are now inserted as parameters into the Python module.

   import argparse  # parse arguments parser = argparse.ArgumentParser(description='Process a DataPower XML Management request.')  parser.add_argument('--username',required=True) parser.add_argument('--password',required=True)  parser.add_argument('--dp_host',required=True) parser.add_argument('--dp_port',required=True) parser.add_argument('--dp_username',required=True) parser.add_argument('--dp_password',required=True) parser.add_argument('--dp_new_domain',required=True)  args = parser.parse_args()  # select target VM and script package vsp_name = 'DataPower - Control VM' vsp_instance_index = 0 vm_index = 0 script_index = 0   # set up script parameter keys dp_host_key = 'DP_HOST' dp_port_key = 'DP_PORT' dp_username_key = 'DP_USERNAME' dp_password_key = 'DP_PASSWORD' dp_new_domain_key = 'NEW_DOMAIN'  # execute script vm = deployer.virtualsystems[vsp_name][vsp_instance_index]. virtualmachines[vm_index] script = vm.scripts[script_index] script_parms = {dp_host_key : args.dp_host, dp_port_key : args.dp_port, dp_username_key :   args.dp_username, dp_password_key : args.dp_password, dp_new_domain_key : args.dp_new_domain} parms = {'script': script, 'username' : args.username, 'password' : args.password, 'parameters' :   script_parms} vm.executeScript(parms)

To invoke this module, you use the CLI client program, passing in the appropriate parameters as follows:

   \pure.cli\bin\pure -h 172.18.104.32 -u someuser -p somepassword -f createDomainCLI.py --username root --password  nnn --dp_host 172.17.109.116 --dp_port 5550 --dp_username admin --dp_password mmm --dp_new_domain test99 

Replace the user ID and password values as required for your environment. You can use the same type of program for all the script packages for DataPower configuration, with minor adjustment to the parameter list, of course.

Making your pattern high availability ready

So far in this article, we have assumed that you want to deploy and configure a single virtual DataPower appliance. In a real test or, even more so, a production environment, you will want to deploy multiple appliances for load balancing and high availability. Given that we can reuse all of the techniques we have described already, you will see how easy it is to set up a fully clustered and load balanced set of virtual appliances using the DataPower Application Optimization (AO) feature!

As explained earlier, the non-production images have all of the licensed add-on features enabled except for the Tibco integration feature. Production images, however, require that you install the licensed features that you have purchased. You can install licensed features into the DataPower appliance using the "boot image" DataPower CLI command, or you can install them automatically as part of a script package using the XML Management interface. You must reboot the appliance after a feature is installed.

An example script package is provided with this article to install a licensed feature. The script called "Install Feature" in Figure 3 downloads a "scrypt4" from a provided URL, base64 encodes it, sends the encoded scrypt4 to the appliance, and then uses it to upgrade the appliance. There is also a script for rebooting the appliance called "Reboot", which is a required operation before you can use the AO feature

You can use the AO feature by creating a virtual system pattern that contains more than one appliance. In this example, we will assume that you want to deploy two appliances, so that there is a backup in case one of them fails. However, keep in mind that you can easily deploy more than two appliances, and you simply configure them to all be part of the same standby group.

Figure 6 shows such a simple pattern, with two appliances, both having their extra NICs removed.

Figure 6. Multiple virtual appliances in one pattern

Next, you have to configure them to be part of the same standby group. When running in this mode, the appliances support two functions:

• Using a common virtual IP (VIP) address, they automatically balance the load between themselves. Requests sent to this VIP are distributed to both appliances according to the defined settings.
• If one of the appliances fails, the other one (or, in the case of multiple appliances, the remaining ones) will automatically take over the load from the failed instance, thus providing continuous availability.

Establishing a virtual IP address

1. Create a new system pattern with a "Core OS" part in it. Deploy that pattern into the same IP group as the DataPower pattern, ensuring that the IP address assigned to the instance is in the same subnet (assuming that the IP addresses in the IP group are all in the same subnet, of course). Once the new instance has been deployed, stop the virtual machine. You can now use the IP address that was assigned to the stopped virtual machine and use it as the VIP for our DataPower standby group, as shown in Figure 7.

   Figure 7. A stopped pattern instance used to establish a virtual IP address

2. You can explicitly remove a valid IP address from the IP group. You can easily do this via the PureApplication System console, as shown in Figure 8.

   Figure 8. Manually removing an IP address from the IP group

This ensures that the address (in our example, 9.5.84.136) is no longer available to be assigned to a virtual machine, so that it can be used as the VIP for our DataPower standby group.

In both cases, record the IP addresses that you have made available, because you will have to define it when configuring the standby group.

Defining a standby group

To configure the virtual appliances to form a standby control group, use the same mechanism as before for the basic configuration: execute a script package that exists on the "DataPower Control VM". The script package is named "Set Standby Control" and its parameter list looks like what is shown in Figure 9.

Figure 9. Parameters for the "Set Standby Control" script package

Besides defining the target for the configuration, such as the DataPower appliance that will contain the standby definition, you define the following parameters:

• AO_INTERFACE: This is the name of the interface to configure for standby control.
• AO_GROUP: This is the group number for the standby group. All appliances that are part of the same cluster get the same number. This number must be unique for groups sharing the same subnet.
• AO_VIRTUALIP: This is the virtual IP address under which the cluster will be accessible for clients.
• AO_PRIORITY: The priority is used to determine where traffic is sent. The priority has to be unique for each member of the cluster.

Additional definitions are available on the interface, for example the algorithm used to distribute incoming traffic. In our example script, we have hard-coded this to "Round Robin".

Note that you have to run this script package against every appliance that you want to be added to the standby control group. Whether these appliances are part of the same pattern instance or not is not relevant for this.

In the DataPower admin console, the completed standby definition looks like Figure 10.

Figure 10. DataPower Standby control configuration

Once you have configured all DataPower appliances that you want to be part of the same standby group, traffic can be directed to the virtual IP address shared by all of them. This traffic is distributed automatically across all members of the group. If one of the appliances has a failure, the others automatically take over.

Conclusion

This article introduced a new way of using the IBM WebSphere DataPower appliance, namely running it as a virtual appliance within PureApplication System. This virtual appliance supports the same set of functionality you get with the physical appliance, just in a different form factor. Moreover, its support for PureApplication System lets you include DataPower appliances in virtual system patterns, which means you can automate its install, configuration, and integration with other parts of your infrastructure.

We have shown you some examples of how you can automatically configure your virtual appliance using script packages that use DataPower's XML Management interface. The script packages are hosted on a separate "control" virtual machine that can execute admin requests against any virtual DataPower appliance deployed on the system.

Finally, we described how to create a highly available topology containing more than one appliance. Multiple appliances are logically joined together by using the Application Optimization feature, which provides functions to create a cluster of self-balancing appliances that support automatic failover in case an appliance fails.

Note that all of the introduced patterns and script examples are available to download with this article.

Acknowledgements

The authors would like to thank Andrew Grohman for reviewing this article and for providing a Python script that calls the DataPower XML Management Interface.