Extending WebSphere DataPower with centralized appliance management

Alexey Roytman, Research staff member, Systems Documentation, Inc. (SDI)

Alexey Roytman is a research staff member at IBM Haifa Research Lab, has spent the last nine years on Web development and architecture tasks, and has authored several technical articles.

Gal Shachor, Senior Technical Staff Member, IBM

Gal Shachor is an IBM Senior Technical Staff Member and researcher working at the IBM Haifa Research Lab, on various topics related to middleware and rich Internet applications. Gal is the author of the book JSP Tag Libraries and several technical articles.

Ben Wen, Product Manager, Systems Documentation, Inc. (SDI)

Ben Wen is a Product Manager at IBM for the line of WebSphere DataPower products. He has held positions in enterprise sales, channel development, engineering, and venture capital. Ben has an EECS degree from MIT.

Gennady Laventman, Research staff member, Systems Documentation, Inc. (SDI)

Gennady Laventman is a research staff member at IBM Haifa Research Lab with expertise in Web Services, Rational software, and tool development.

Summary: IBM® WebSphere® DataPower® SOA Appliances are initially configured using an interactive Web GUI-based console. Operational interfaces like IBM Tivoli® Composite Application Manager System Edition push XML-encoded rules into one or more appliances in a cluster. Automated scriptable interfaces and Web services endpoints also enable rich management. This article presents an alternative approach in which simple rules are stored outside the appliances and interpreted by them on demand. Using some common Web and Web services patterns, this approach similarly enables management when several appliances are in use. By directing message routing rules to an external, shared storage location, this technique can simplify the task of appliance management by minimizing changes made to each device, by centralizing more frequently changed metadata, and by assisting integration with third party tools. This technique can be generalized beyond routing rules to other uses as well. This content is part of the IBM WebSphere Developer Technical Journal.

Tag this!

Date: 03 Sep 2008
Level: Intermediate
Activity: 1234 views
Comments:

Introduction

IBM WebSphere DataPower SOA Appliances represent an important element in Service Oriented Architecture (SOA). The appliances are purpose-built, easy-to-deploy network devices that simplify, secure, and accelerate XML and Web services deployments while extending an SOA infrastructure.

One simple example for its use is a multi-service environment in which inbound requests can be sent into one of several Web service providers. In such a scenario, an incoming service request is evaluated based on the incoming parameters (URL, headers, and request parameters) and system status (date, load, and so on).

In a relatively static use case, the routing rules would not change very often. Routing rules can be configured manually using a Web GUI-based console or by uploading XML-encoded rules. While authoring the routing rules inside the appliance is not a demanding task, managing consistent configuration across a cluster of appliances increases operational effort.

IBM provides a multi-box management solution in IBM Tivoli Composite Application Manager System Edition (ITCAM SE), which enables administrators to efficiently roll out, update, and manage configurations across multiple WebSphere DataPower SOA Appliances. ITCAM SE pushes configuration to each device in turn. Further, IBM WebSphere Registry and Repository is also available for enterprise-class centralized policy storage and management. The DataPower devices can dynamically consume policies made available in WebSphere Registry and Repository for compliant Web services traffic.

About this approach
The techniques shown in this paper are applicable for rapid prototyping, ad hoc development projects, non-standards-based implementations, or incremental deployments on the path to more complete standards. Other usage could be contemplated for applications where more heavyweight standards might not be applicable or desirable. For actual production applications, you should investigate standards (such as WSDL, WS-Addressing, UDDI, WS-Notification, and so on) and products (such as ITCAM SE and WebSphere Registry and Repository) to determine their applicability for your particular use case.

This article presents a pull-based approach that can apply to both the WebSphere DataPower XML Security Gateway XS40 and the WebSphere DataPower Integration Appliance XI50. With this approach, you create a centralized management solution by configuring generic routing logic into each appliance and then have the script call external routing rule services. The rule services resolve the actual route lookup for the generic logic. This technique enables the routing rules to be managed in a single centralized repository used by several appliances. The appliances can cache route lookups so that subsequent traffic does not require an external network transaction. This method works with XML, Web services, and non-XML traffic. In this approach, you implement a simple two-step lookup to handle a class of cache staleness.

The rich, flexible, standards-based customization capabilities of WebSphere DataPower appliances enable you to easily fit the appliance in many situations. Through this demonstration, you will see the strength of the DataPower APIs and be encouraged to further investigate the rich set of configuration objects, extensions, and elements that are available. While you work with standard operating practices as they evolve and expand to fit more use cases, you should be pragmatic in utilizing customization capabilities to solve problems today

The remainder of this article presents the various steps associated with developing the routing scripts and configuring them in the DataPower appliance, starting with a brief system overview and a description of the routing rules, followed by steps to configure the appliance and develop the routing scripts.

System overview

Figure 1 shows an edited presentation of the system (omitting components such as firewalls) in which the appliance and routing scripts operate in this scenario. In general, the system is composed of:

Several clients that access services though the network.
Servers hosting the various services used by the clients.
An external central routing rule repository, exposed through an HTTP URL.
One or more WebSphere DataPower SOA Appliances that route between the clients and their servers. Inside this appliance are routing scripts that communicate with the central rule repository to make decisions.

Figure 1. System overview

Routing a certain client to a service involves these steps:

The route script running inside the appliance fetches a routing rules list from the remote Web server repository. The rules exist as an XML document, and, for improved performance, can be cached locally by the DataPower appliance with settable time-to-live.
The request parameters and, especially, destination URL, are matched with the rules’ patterns.
As a backup, if a destination was not resolved (perhaps due to caching staleness), a Web services request is sent to a routing rules resolution server (the routing rules repository and the routing rules Web service can be hosted on the same or different servers). The request contains the request URL as key parameters. The URL-based key can be easily expanded to include other characteristics of the user request.
If the destination was resolved, the original user request is forwarded to the destination; otherwise an error is returned to the user.

Routing rules and repository server

The route rules employed here are encoded using an XML format, as shown on Listing 1. Each rule contains:

Rule name: unique name that is used for the logging and rule management.
URL pattern: a Perl-compatible regular expression string to be matched with the URL sent by the client.
Destination URL: the target service URL.

Listing 1. Rules.xml

<?xml version="1.0" encoding="UTF-8"?> 
<rules>
    <rule>
        <name>rule1</name>
        <pattern>http://.*/first/.*</pattern>
        <destination>
           http://rhel3-6.haifa.ibm.com:9080/TestWeb1/services/TestSoap
        </destination>
    </rule>
    <rule>
        <name>rule2</name>
        <pattern>.*second.*</pattern>
        <destination>
           http://rhel3-6.haifa.ibm.com:9080/TestWeb2/services/TestSoap
        </destination>
    </rule>
</rules>

Following the rules encoded in Listing 1:

The URL http://test-lnx.asdf.com/first/test will be routed to http://rhel3-6.haifa.ibm.com:9080/TestWeb1/services/TestSoap.
The URL http://www.secondTest.com/hello/ will be routed to http://rhel3-6.haifa.ibm.com:9080/TestWeb2/services/TestSoap.
The URL http://www.ibm.com/ will not be matched, and will be rejected.

The routing rules are stored in a rules repository with two interfaces:

Simple GETFull HTTP-based interface: a client can send an HTTP GET request and download a list of all rules; the output of such a request will follow the format presented in Listing 1.
SOAP-based interface with a single method named getWSDestinationForUrl: the method accepts the key (URL) as its single parameter, and returns a destination URL or null. (The Web Service WSDL and examples of the code are available in the included download file.)

Configuring an XML firewall

The first step in using the routing script is to configure a WebSphere DataPower XMLFirewall object on your appliance.

Using the WebSphere DataPower control panel, create a new XML firewall called “dw-example,” with a Firewall Type of dynamic-backend, Response Type of Pass-Thru, and Request Type of Non-XML (other request types like XML and SOAP are suitable as well, but enforce XML well-formedness, and SOAP enveloping and schema, respectively). The dynamic-backend type is important because it informs the device that the backend host and URL are subject to modification by the processing policy. This is what enables the generic routing logic to do its work.

The firewall definition UI is shown in Figure 2.

Figure 2. XML firewall

XML firewall policy definition

Next, create a new firewall policy with one request processing rule. This rule contains a fetch action and three transformation actions that implement your routing logic. All transformation actions can be combined into one, but they are split up in this example for readability purposes. The policy definition UI is shown in Figure 3. Notice that the very first icon in the policy is a disambiguation “match” action that is required by the device to decide between multiple rules. Because there is only one rule here, the match action is not used.

Figure 3. Configure XML firewall policy

Fetch action

The fetch action is used to get the entire set of routing rules from the repository and place them in a temporary WebSphere DataPower context. The fetch action is a predefined WebSphere DataPower action, with one source parameter that you point to the routing rules repository URL. Like its name suggests, this fetch action makes an HTTP request to fetch the configured source URL. At this point, the fetched document is generic and has no meaning to the device.

Additionally, to avoid excessive traffic to the remote repository, you can use the WebSphere DataPower provided cache, as can be managed from the XML Manager object associated in the previous XML firewall screen and described in the XML Manager definition section.

The output “tempvar1” will contain the fetched XML document. The fetch action definition UI is shown in Figure 4.

Figure 4. Configure fetch action

First route resolution action

The next action is a WebSphere DataPower transformation that executes the first route resolution stage. The XSLT script used for the transformation (Listing 2) is stored locally on the appliance, although it could also be stored remotely for even further cluster manageability. The script uses a WebSphere DataPower context variable to pass its result to the following actions. The actual transformation output generated by the script is unused because the setting of the context variable is essentially a side effect to the transform, and is the real routing work that we are interested in. The transformation definition UI is shown in Figure 5.

Figure 5. Configure transform action

The script goes over all the routing rules that were received in the prior fetch action (tempvar1 is used as the input to this action), and tries to match the inbound messages’ URL with the URL pattern in the various rule entries, using regular expressions. If a certain rule is matched, the rule name and its destination are stored in WebSphere DataPower context variables var://context/dw/route and var://context/dw/ruleName. The rule name can be logged for audit and troubleshooting reasons. (An XSLT for-each element does not have a break option, so if several rules are matched, the last one will be used.)

Listing 2. Rules processor XSL file

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
	xmlns:dp="http://www.datapower.com/extensions"
	xmlns:dpconfig="http://www.datapower.com/param/config"
	xmlns:regexp="http://exslt.org/regular-expressions"
	extension-element-prefixes="dp"
	exclude-result-prefixes="dp dpconfig regexp">

   <xsl:template match="rules">
      <xsl:for-each select="rule">
         <xsl:variable name="pattern">
	      <xsl:value-of select="pattern" />
         </xsl:variable>

         <xsl:if test="regexp:test(dp:variable('var://service/URL-in'),
			normalize-space($pattern))">
            <xsl:variable name="dest">
		   <xsl:value-of select="destination" />
		</xsl:variable>
		<dp:set-variable name="'var://context/dw/route'"
						value="normalize-space($dest)" />
		<dp:set-variable name="'var://context/dw/ruleName'"
						value="string(name)" />
         </xsl:if>
      </xsl:for-each>
   </xsl:template>
</xsl:stylesheet>

Second route resolution action

The next transformation action uses another XSLT script to query the rule server, in case of a miss in the previous stage. The input is not used (an empty XML document) and the output is ignored. The XSLT document used by the action is using a single parameter, managerURL, which is the route-resolving Web service URL.

The first activity performed by the action’s XSLT script is to check for an existing routing decision in the context variable, var://context/dw/route. If the routing was not resolved, a SOAP request is created and sent to the secondary route-resolving Web service:

The Web service request has one input parameter, the request URL.
The response contains the destination URL, or null in case of failure.

The discovered destination is stored in the same WebSphere DataPower context variable, var://context/dw/route. The action XSLT file is shown in Listing 3.

Figure 6. Second route-resolving action

Listing 3. Second route-resolving XSL file

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
	xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
	xmlns:dp="http://www.datapower.com/extensions"
	xmlns:dpconfig="http://www.datapower.com/param/config"
	xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/"
	xmlns:ejbs="http://ejbs" extension-element-prefixes="dp"
	exclude-result-prefixes="dp dpconfig SOAP">

<!--  URL of the Manager,  e.g. http://lnx-dw.haifa.ibm.com:9081/RouterManagerWEB/
	services/WebServiceDestinationSOAPRouting -->

   <xsl:param name="dpconfig:managerURL" />
   <xsl:template match="/">
   <xsl:if test="not(dp:variable('var://context/dw/route'))">
      <xsl:variable name="call">
         <SOAP:Envelope
                xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <SOAP:Body>
               <ejbs:getWSDestinationForUrl>
                  <pUrl>
                     <xsl:copy-of 
                         select="dp:variable('var://service/URL-in')"/>
                  </pUrl>
               </ejbs:getWSDestinationForUrl>
            </SOAP:Body>
            </SOAP:Envelope>
         </xsl:variable>

         <xsl:variable name="result"
                  select="dp:soap-call($dpconfig:managerURL, $call)" />
			
         <xsl:if test="nilled($result)">
            <dp:set-variable name="'var://context/dw/route'"
					value="string($result)" />
         </xsl:if>
      </xsl:if>
   </xsl:template>
</xsl:stylesheet>

Route definition action

The last transformation action executes the XSLT file script shown in Listing 4. Once again, the action checks that var://context/dw/route was defined by either of the previous route actions, and sets WebSphere DataPower service variables var://service/routing-url and var://service/URI accordingly. The setting of these two special variables triggers a redirect to a URL wherein var://service/routing-url defines the protocol/host/port portion and var://service/URI sets the destination path. (For a full list of special variables, see the WebSphere DataPower documentation in Resources.) If a destination was not found, the action sets an error message and rejects the request.

Once this transformation action completes, the final action is a results action, which informs the device that the processing rule has completed and the message can be sent onto is (new) destination.

Listing 4. Route definition XSL file

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
	xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
	xmlns:dp="http://www.datapower.com/extensions"
	xmlns:dpconfig="http://www.datapower.com/param/config"
	extension-element-prefixes="dp"
	exclude-result-prefixes="dp dpconfig">

	<xsl:template match="/">
		<xsl:variable name="destination"
			select="dp:variable('var://context/dw/route')" />

		<xsl:choose>
			<xsl:when test="$destination">
				<dp:set-variable name="'var://service/routing-url'"
					value="$destination" />
				<dp:set-variable name="'var://service/URI'"
					value="$destination" />
			</xsl:when>
			<xsl:otherwise>
				<dp:set-variable name="'var://context/dw/errorMessage'"
					value="'Undefined rule'" />
				<dp:reject override="true">Undefined rule</dp:reject>
			</xsl:otherwise>
		</xsl:choose>
	</xsl:template>
</xsl:stylesheet>

XML Manager definition

To prevent numerous fetch requests, we can configure an XML Manager to cache the fetch action results. According to the definition in Figure 8, all documents from lnx-test.haifa.ibm.com will be cached for 900 seconds.

Figure 7. Caching configuration

Summary

The approach presented in this article uses the WebSphere DataPower extension mechanisms to provide a high performance, externally-managed, and flexible routing mechanism. Although the URL is used as the sole routing parameter in the context of this article, other parameters can also be used. For example, you could instead base the routing decision on information from the inbound request payload (parameters) and headers.

While this article showcased an ad-hoc multi-box management technique, IBM Tivoli Composite Application Manager System Edition provides a far more robust management solution that can also integrate easily with the rest of your management infrastructure.

Resources

Learn

Get products and technologies

Download WebSphere DataPower SOA Appliances firmware and documentation

About the authors

Alexey Roytman is a research staff member at IBM Haifa Research Lab, has spent the last nine years on Web development and architecture tasks, and has authored several technical articles.

Gennady Laventman is a research staff member at IBM Haifa Research Lab with expertise in Web Services, Rational software, and tool development.

Comments

Trademarks | My developerWorks terms and conditions

Close [x]

Help: Update or add to My dW interests

What's this?

This little timesaver lets you update your My developerWorks profile with just one click! The general subject of this content (AIX and UNIX, Information Management, Lotus, Rational, Tivoli, WebSphere, Java, Linux, Open source, SOA and Web services, Web development, or XML) will be added to the interests section of your profile, if it's not there already. You only need to be logged in to My developerWorks.

And what's the point of adding your interests to your profile? That's how you find other users with the same interests as yours, and see what they're reading and contributing to the community. Your interests also help us recommend relevant developerWorks content to you.

View your My developerWorks profile

Return from help

Close [x]

Help: Remove from My dW interests

What's this?

Removing this interest does not alter your profile, but rather removes this piece of content from a list of all content for which you've indicated interest. In a future enhancement to My developerWorks, you'll be able to see a record of that content.

View your My developerWorks profile

Return from help

static.content.url=http://www.ibm.com/developerworks/js/artrating/

SITE_ID=1

Zone=WebSphere

ArticleID=334415

ArticleTitle=Extending WebSphere DataPower with centralized appliance management

publish-date=09032008

author1-email=roytman@il.ibm.com

author1-email-cc=

author2-email=shachor@il.ibm.com

author2-email-cc=

author3-email=benwen@us.ibm.com

author3-email-cc=

author4-email=gennady@il.ibm.com

author4-email-cc=

Introduction
System overview
Routing rules and repository server
Configuring an XML firewall
Summary
Resources
About the authors
Comments

Next steps from IBM

Rational Tester for SOA Quality is an automated web services testing and SOA testing tool that supports Web services SOAP, HTTP(S), JMS, WS-Security, UDDI, and RESTful standards.