Processing attachments in WebSphere DataPower SOA Appliances

This article shows you how to use WebSphere DataPower SOA Appliances to offload and process Web services attachments without compromising overall performance. Since the appliances are in the DMZ, they can analyze attachments and check for viruses and other threats. If a threat is found, the attachment can be quarantined or striped, the rest of the message can be processed, and the attachment can be analyzed later.

Share:

Andrew A. Das (aadas@us.ibm.com), Senior IT Specialist, IBM

Andrew Das photoAndrew A. Das is a Senior IT Specialist for WebSphere Datapower SOA appliances. In his prior role, he was part of the WebSphere Application Server Web Services Interoperability development team, where he was responsible for evaluating Web services standards and their interoperability with IBM products and other vendor products. You can reach Andrew at aadas@us.ibm.com.



Andy Thurai (andythurai@gmail.com), Director, L1 Identity Solutions

Photo of Andy ThuraiAndy Thurai is a Senior Director, Enterprise Architecture with L1 Identity Solutions (www.L1id.com). His interests and expertise include SOA, identity management, security, governance, cloud computing, and SaaS. He holds a degree in Electrical and Electronics engineering and has over 20 years of IT experience with firms such as IBM, CSC, BMC, and Nortel. You can contact Andy at andythurai@gmail.com.



29 September 2010

Introduction

IBM® WebSphere® DataPower® Appliances are well suited as intermediaries in the DMZ playing the role of Web Services proxies. DataPower Appliances are purpose-built hardware platforms that provide a configuration-driven approach to process SOAP attachments with the added benefit of reduced transactional latency due to their patented technology and hardware characteristics when compared to any software based alternative.

This article shows you how to use WebSphere DataPower to offload and process Web Services attachments without compromising overall performance. Creating policies to serialize and de-serialize binary content within SOAP message is radically simpler in DataPower due to its graphical configuration, though this article provides a more complex scenario to demonstrate DataPower flexibility. The article focuses primarily on interactions based on SwA and MTOM, as these are the most common ways to transmit binary content over SOAP. It will also show how content can be serialized and de-serialized to base64, which happens to part of the configuration and will give you some insight when you are dealing with base64 content manipulation.

Overview of Web services attachments

With the growing popularity of Web services as the fundamental messaging paradigm for SOA, there has been a need to transmit binary data in cases where data cannot be expressed in a structured format like XML. XML and hence SOAP cannot embed the binary messages inside the tags like CDATA elements. Consider a scenario where a customer deposits a check at an ATM and an image of the check must be transferred to a back-end system for further processing, or a case where a physician needs to transfer scanned copies of patient records to another physician. Traditionally, such scenarios were handled by using programming APIs to convert images, PDFs, or any binary content to base64 and then embedding that within a SOAP message.

Because of several drawbacks to that approach, competing technologies emerged from Web services vendors, in parallel with specifications proposed by WS standards bodies. During the course of this evolution, many of these technologies were discarded, merged, or improved upon, resulting in a handful of popular techniques and standards. Based on popular choice and wide support, three standards emerged: base64 programmatic encoding, SOAP with Attachments (SwA), and Message Transmission Optimization Mechanism (MTOM). Although MTOM is considered superior (as described in the section below), the other two standards existed for years prior to MTOM and therefore are still used due to their widespread deployment and familiarity. The DataPower team strives to address all customer requirements by supporting all three of these standards. In fact, the DataPower team goes a step further with support for Direct Internet Message Encapsulation (DIME), an alternative to SwA proposed by Microsoft to provide a completely heterogeneous and interoperable platform for attachment processing.

Comparison of Base64, SwA, and MTOM

base64Binary
The attachment is sent as base64 inline and embedded in the SOAP message, which expands the message by 33% and requires increased memory and CPU time.
SWA
The SOAP message contains a reference to the attachment, which is outside of the message. But because the SOAP infoset does not contain the attachment, custom handling is required when interacting with other WS- specifications.
MTOM
The best of both worlds: the SOAP message contains a reference to the attachment, which is sent outside the SOAP message, but the attachment appears as if it is embedded inside the SOAP message, and the SOAP infoset contains the attachment. This solution is compatible with WS- specifications and is highly interoperable.

Scenario

Figure 1 shows the scenario used to demonstrate the conversion between SwA and MTOM. SwA clients send a SOAP request to DataPower with binary content (such as a .jpg image) attached in compliance with the SwA specification. Assuming that the service provider is an MTOM provider, you would need a mechanism to do this conversion. A Web service proxy service defined within DataPower will perform the necessary operations to convert SwA to MTOM. Once the MTOM provider processes the MTOM payload from DataPower, the provider sends a response back with a different attachment (a different .jpg image), which is MTOM-compliant. The Web service proxy converts the MTOM payload back to SwA format for client consumption, which enables legacy clients to continue to consume SwA attachments while the service providers can develop new MTOM-based services.

Figure 1. Scenario
Scenario

Prerequisites

  • SOAP UI (optional) – Testing tool
  • Curl (optional) – Testing tool
  • WebSphere DataPower SOA Appliance (XS40, XI50, XB60, or XM70)
  • Device configuration and other artifacts from the files which you can download at the bottom of the article.

Configuration

This section shows you how to configure DataPower to process attachments embedded in SOAP messages transported over HTTP.

Setting up the Web services proxy (WS-Proxy)

Follow these steps to create the WS-Proxy if you did not import the pre-configured Web services proxy:

  1. From the DataPower control panel, select Web Service Proxy => Add.
  2. Enter SWA_MTOM_Mediation_WSP for the name and select Create Web service proxy.
  3. Upload Attachments.wsdl from the download file, select Off for WS-Policy References, and then click Next:
    Figure 2. WS-Proxy configuration
    WS-Proxy configuration
  4. Select the + under Local Endpoint Handler and HTTP Front Side Handler.
  5. Enter http-9080 for the Name and 9080 for the Port Number, and then click Apply.
  6. Back on the WS-Proxy configuration panel, select Add under Edit/Remove, and then Next at the bottom.
  7. Save your configuration. You have completed the setup of the WS-Proxy.
  8. Click on the Policy tab, and then click Operations to view the Service Editor. If you imported the sample, you will notice the pre-configured actions at the port-operation level, as shown in Figure 3 below. At the operation level there are also three red crosses indicating that SOAP request, response, and fault checking is disabled:
    Figure 3. Web Service Proxy Policy – rules and actions
    Web Service Proxy Policy – rules and actions
  9. Ensure that the correct service attachment mode is selected for both request and response. Go to the DataPower control panel and click on Objects => Web Service Proxy. Click on SWA_MTOM_Mediation_WSP (unless you created the WSP with a different name). Select Proxy Settings and scroll down to the drop-down for Request Attachment Processing Mode and Response Attachment Processing Mode. You will see five different values; change both Request and Response to Allow:
    • Allow – Processes the message root and needed XML and non-XML attachments. Needed attachments are buffered. Attachments that are not needed might be streamed directly to output.
    • Reject – Discards messages that contain attachments.
    • Streaming – Provides limited processing of XML attachments, and streams XML and non-XML attachments to output.
    • Unprocessed – Allows messages that contain attachments, but does not process attachments.
    • Strip – (Default) Removes attachments from the message and changes the value of the Content-Type header to that of the root part.
    Figure 4. Web services proxy object – attachment processing mode
    Web services proxy object – attachment processing mode

    Click to see larger image

    Figure 4. Web services proxy object – attachment processing mode

    Web services proxy object – attachment processing mode
  10. Finally, an MTOM policy global object needs to be created to serialize the base64 binary data message to MTOM. If you are processing multiple attachments, then each attachment in the multi-part message will have a corresponding XPath entry and an optional Content-Id to identify it. Select Encode under the Main tab of the MTOM Policy object (The same object can also be used to de-serialize MTOM to base64 binary inline data by selecting Decode). You will also be required to enter a unique name. To create an MTOM Policy go to Objects => MTOM Policy under XML Processing=. Click on EncodeToMTOM to view the pre-created object. The object should include an entry under MTOM Rules that contains an XPath entry value //*[local-name()='binaryData'] with an associated Content-ID value bluehills, as shown in Figure 5 below:
    Figure 5. MTOM policy object configuration
    MTOM policy object configuration

SwA => MTOM (request rule)

Listing 1. Ingress SOAP (SwA):
POST /MTOMSample HTTP/1.1
SOAPAction: "attachmentSwA"
Content-Type: multipart/related; type="text/xml"; start="<rootpart@soapui.org>"; 
  boundary="----=_Part_12_22670407.1276619880828"
  MIME-Version: 1.0
  User-Agent: Jakarta Commons-HttpClient/3.1
  Host: 9.42.90.61:1111
  Content-Length: 29265            
  
  ------=_Part_12_22670407.1276619880828 Content-Type: text/xml; 
  charset=UTF-8 Content-Transfer-Encoding: 8bit
  Content-ID: <rootpart@soapui.org>
    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
      xmlns:swa="http://ws.apache.org/axis2/mtomsample/" xmlns:xm=
          "http://www.w3.org/2005/05/xmlmime">   
      <soapenv:Header/>   
      <soapenv:Body>      
        <swa:AttachmentRequest>         
          <swa:fileName>bluehills.jpg</swa:fileName>         
          <swa:binaryData>cid:bluehills</swa:binaryData>      
        </swa:AttachmentRequest>   
      </soapenv:Body>
    </soapenv:Envelope>------=_Part_12_22670407.1276619880828
    Content-Type: image/jpegContent-Transfer-Encoding: binaryContent-ID: 
    <bluehills> ******BINARY DATA TRUNCATED***********
Listing 2. Egress SOAP (MTOM):
POST /mockSwAServiceSOAP11Binding HTTP/1.1
SOAPAction: "attachmentSwA"
Content-Type: multipart/related; boundary="2e04f680-8d9a-4c55-b331-6866af11437c"; 
start-info="application/soap+xml"; type="application/xop+xml"
MIME-Version: 1.0
User-Agent: Jakarta Commons-HttpClient/3.1
Host: 127.0.0.1:9090
Via: 1.1 AQAAAAEAAAA=
X-Client-IP: 9.65.90.148
Connection: Keep-Alive
Content-Length: 29378

--2e04f680-8d9a-4c55-b331-6866af11437c
Content-Type: application/xop+xml; type="application/soap+xml"
Content-Transfer-Encoding: binary

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:xm="http://www.w3.org/2005/05/xmlmime" 
  xmlns:swa="http://ws.apache.org/axis2/mtomsample/" 
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header/>   
  <soapenv:Body>      
    <swa:AttachmentRequest>
      <swa:fileName>bluehills.jpg</swa:fileName>
      <swa:binaryData>
        <xop:Include href="cid:bluehills" 
          xmlns:xop="http://www.w3.org/2004/08/xop/include"/>
      </swa:binaryData>      
    </swa:AttachmentRequest>   
  </soapenv:Body>
</soapenv:Envelope>--2e04f680-8d9a-4c55-b331-6866af11437cContent-ID: 
<bluehills>Content-Transfer-Encoding: binaryContent-Type: application/binary 
  *****BINARY DATA TRUNCATED*****

Actions: The request rule of the service (port-operation level) policy contains actions (configurations) shown in Figure 6 below:

Figure 6. Web services proxy – request rule configuration
Web services proxy – request rule configuration
  1. Match -- Triggers the rule to match any URL specified by the incoming request.
  2. Transform -- References the XSLT that converts the incoming SwA attachment to base64 binary.
    Listing 3. getAttachments.xsl
    <?xml version="1.0" encoding="utf-8"?>
    <!-- This stylesheet finds the incoming attachment(s), serializes them to base64, 
      and embeds them in the XML infoset -->
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:dp="http://www.datapower.com/extensions"
      xmlns:swa="http://ws.apache.org/axis2/mtomsample/"
      xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
      xmlns:dpconfig="http://www.datapower.com/param/config" 
        extension-element-prefixes="dp"
      exclude-result-prefixes="dp dpconfig">
      
      <xsl:template match="//swa:binaryData">
        <xsl:copy>
          <xsl:copy-of select="*"/>
          <!-- This pulls the manifest variable from the INPUT context  from this  -->
          <xsl:variable name="manifest"
            select="dp:variable('var://context/INPUT/attachment-manifest')"/>
          
          <!-- Create a variable that is the name of the attachment and loop thru getting
            all attachments. Multiple attachments can be handled thru this loop -->
          <xsl:for-each select="$manifest/*[local-name()='manifest']/
              *[local-name()='attachments']/*">
            <xsl:variable name="filename" select="./uri/text()"/>
            <xsl:variable name="cid" select="concat($filename,'?Encode=base64')"/>
            <xsl:variable name="elementname" select="substring-after($filename,'cid:')"/>
            <xsl:variable name="base64"><dp:url-open target="{$cid}"/></xsl:variable>
            <xsl:value-of select="$base64/base64" />
          </xsl:for-each>
        </xsl:copy> 
      </xsl:template>
      
      <xsl:template match="*|@*|node()">
        <xsl:copy>
          <xsl:apply-templates select="@*"/>
          <xsl:apply-templates/>
        </xsl:copy>
      </xsl:template>
    </xsl:stylesheet>
  3. Strip-Attachments -- Removes the binary content from the MIME headers and adjusts the HTTP headers accordingly.
  4. Validate (Optional)-- Encapsulates a link to the WSDL file (Attachments.wsdl) against which the SOAP request needs to be validated. If this step fails, the transaction will terminate before forwarding the message to the back end.
  5. Transform -- References the pre-shipped mtom.xsl file in the store:/// directory. Once this stylesheet is selected, the MTOM policy dropdown appears. Select the MTOM policy that you created earlier:
    Figure 7. Associate MTOM configuration in Transform action
    Associate MTOM configuration in Transform action

MTOM => SwA (Response rule)

Listing 4. Ingress SOAP (MTOM):
HTTP/1.1 200 OK
Content-Type: multipart/related; type="application/xop+xml"; 
  start="<rootpart@soapui.org>"; 
  start-info="text/xml"; boundary="----=_Part_19_32674460.1276620855718"
  MIME-Version: 1.0
  Transfer-Encoding: chunked
  Server: Jetty(6.1.x)
  
  ------=_Part_19_32674460.1276620855718
  Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"
  Content-Transfer-Encoding: 8bit
  Content-ID: <rootpart@soapui.org>
    
    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
      xmlns:swa="http://ws.apache.org/axis2/mtomsample/">
      <soapenv:Header/>
      <soapenv:Body>
        <swa:AttachmentResponse>
          <swa:fileName>waterlillies.jpg</swa:fileName>
          <swa:binaryData><inc:Include href="cid:waterlillies" 
            xmlns:inc="http://www.w3.org/2004/08/xop/include"/></swa:binaryData>
        </swa:AttachmentResponse>
      </soapenv:Body>
    </soapenv:Envelope>
    ------=_Part_19_32674460.1276620855718
    Content-Type: image/jpeg
    Content-Transfer-Encoding: binary
    Content-ID: <waterlillies>
    *********BINARY DATA TRUNCATED*********
Listing 5. Egress SOAP (SwA):
HTTP/1.1 200 OK
X-Backside-Transport: OK OK
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: multipart/related; boundary="----=_Part_19_32674460.1276620855718"; 
start-info="text/xml"; 
type="text/html"; start="<rootpart@soapui.org>"
  MIME-Version: 1.0
  Server: Jetty(6.1.x)
  Date: Tue, 15 Jun 2010 16:54:14 GMT
  X-Client-IP: 9.65.90.148
  
  ------=_Part_19_32674460.1276620855718
  Content-Type: text/html; charset=UTF-8
  Content-Transfer-Encoding: binary
  Content-ID: <rootpart@soapui.org>
    
    <?xml version="1.0" encoding="UTF-8"?>
    <soapenv:Envelope xmlns:swa="http://ws.apache.org/axis2/mtomsample/" 
      xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
      <soapenv:Header/>
      <soapenv:Body>
        <swa:AttachmentResponse>
          <swa:fileName>waterlillies.jpg</swa:fileName>
          <swa:binaryData>cid:waterlillies</swa:binaryData>
        </swa:AttachmentResponse>
      </soapenv:Body>
    </soapenv:Envelope>
    ------=_Part_19_32674460.1276620855718
    Content-Type: image/jpeg
    Content-Transfer-Encoding: binary
    Content-ID: <waterlillies>
    *******BINARY DATA TRUNCATED**********

Actions: The response rule of the Service (port-operation level) policy contains actions (configurations) illustrated in Figure 8 with descriptions below.

Figure 8. Web services proxy – response rule
Web services proxy – response rule
  1. Match -- Triggers the rule to match any URL specified by the incoming request.
  2. Transform (optional) -- References the XSLT file (getAttachments.xsl) that converts the MIME attachment to base64 and includes it within the message. This is the first ancillary step that is only required to prepare the message for WSDL validation in Step 4.
  3. Transform (optional) -- References the XSLT file that removes the <inc:Include… element from the message. This is the second ancillary step to prepare the message for WSDL validation in Step 4.
    Listing 6. stripInclude.xsl
    <?xml version="1.0" encoding="utf-8"?>
    
    <!--  This stylesheet removes the XOP element, i.e. <inc:Include ....> -->
    
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:dp="http://www.datapower.com/extensions"
      xmlns:swa="http://ws.apache.org/axis2/mtomsample/"
      xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
      xmlns:inc="http://www.w3.org/2004/08/xop/include"
      xmlns:dpconfig="http://www.datapower.com/param/config" 
        extension-element-prefixes="dp"
      exclude-result-prefixes="dp dpconfig inc">
      
      
      <xsl:template match="//inc:Include" />
      
      <xsl:template match="*|@*|node()">
        <xsl:copy>
          <xsl:apply-templates select="@*"/>
          <xsl:apply-templates/>
        </xsl:copy>
      </xsl:template>
      
    </xsl:stylesheet>
  4. Validate (optional) -- In this action, the response SOAP message is validated against the associated WSDL, (Attachments.wsdl).
  5. Transform -- References an XSLT stylesheet that converts the MTOM response to an SwA message.
    Listing 7. setAttachments.xsl
    <?xml version="1.0" encoding="utf-8"?>
    <!-- This stylesheet removes XOP content and modifies the MTOM message 
      to be SwA conforming -->
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:dp="http://www.datapower.com/extensions"
      xmlns:swa="http://ws.apache.org/axis2/mtomsample/"
      xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
      xmlns:inc="http://www.w3.org/2004/08/xop/include"
      xmlns:dpconfig="http://www.datapower.com/param/config" 
        extension-element-prefixes="dp"
      exclude-result-prefixes="dp dpconfig inc">
      
      <xsl:template match="/soapenv:Envelope/soapenv:Body/
          swa:AttachmentResponse/swa:binaryData">
        <xsl:apply-templates select="//inc:Include" />
        <xsl:copy>
          <xsl:variable name="manifest" select=
            "dp:variable('var://context/INPUT/attachment-manifest')"/>
          <!-- Create a variable that is the name of the attachment and loop thru getting
            all attachments. Multiple attachments can be handled thru this loop -->
          <xsl:for-each select="$manifest/*[local-name()='manifest']/
            *[local-name()='attachments']/*">
            <xsl:value-of select="./uri/text()"/> 
          </xsl:for-each>
        </xsl:copy> 
      </xsl:template>
      
      <xsl:template match="//inc:Include" />
      
      <xsl:template match="*|@*|node()">
        <xsl:copy>
          <xsl:apply-templates select="@*"/>
          <xsl:apply-templates/>
        </xsl:copy>
      </xsl:template>
    </xsl:stylesheet>

Import sample

To import the DataPower configuration:

  1. In the DataPower admin console, select Administration => Configuration => Import Configuration.
  2. Click Browse, select MTOM_Domain_Backup.zip from the download file, and then select Next:
    Figure 9. DataPower domain import
    DataPower domain import
  3. SelectImport. At this point your imported configuration is loaded and ready to process traffic. You may have to modify the IP addresses for your environment.

Conclusion

This article described leading Web services attachments and how DataPower supports them. It then walked you through a scenario involving the most widely used DataPower functions related to attachment processing (SwA and MTOM). You should now have an understanding of Web services attachment specifications and how you can configure DataPower for SwA and MTOM processing.

Acknowledgements

The authors would like to thank Hermann Stamm-Wilbrandt and Ulrik Andersen of IBM for providing technical guidance and artifacts used to develop the samples in this article.


Downloads

DescriptionNameSize
Code sampleMTOM_Domain_Backup.zip433 KB
Code sampleSoapUI_Export.zip422 KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=548326
ArticleTitle=Processing attachments in WebSphere DataPower SOA Appliances
publish-date=09292010