Handling Complex Web Services in InfoSphere Information Server through DataStage ASB Packs v2.0

A step-by-step guide

With most organizations and enterprises moving towards Enterprise service-oriented architecture, it is important to look beyond the jargon into what it means for developers and implementers and what exactly are the facilities provided by the various software products and solutions to enable Enterprise SOA based integration. In the current economic and heterogeneous business scenarios involving complex web service applications, IBM InfoSphere Information Server cost effectively addresses the challenges of business integration across enterprises by invoking Complex Web Services with its ASB (Application Services Backbone) Packs.

Santhosh Kumar Kotte (sankotte@in.ibm.com), Systems Software Engineer, IBM

Santhosh is a Systems Software Engineer working for IBM InfoSphere Information Server Group at India Software Lab, Hyderabad. He has joined IBM as an Associate Software Engineer in 2007. He works on the development & maintainance of DataStage ASB Packs(XML Pack 2.0,Java Pack 2.0 and Web Services Pack 2.0) and the next generation Web Services Pack (WS Pack 3.0) development. His primary interests are working with emerging technologies including XML, DB2 pureXML, SOA, web services and exploring various Information Integration Technologies.



24 May 2011

Also available in Chinese

Introduction to IBM InfoSphere Information Server and DataStage ASB Packs v2.0

IBM InfoSphere DataStage ASB Packs v2.0 (XML, Java and Web services Packs) are components of DataStage that helps in delivering faster data integration solutions. This article explores the various features provided by DataStage ASB Packs in invoking Complex Web Services and integrating the data returned by these web services with other DataStage components.DataStage Web Services Pack by itself cannot invoke Complex Web Services. This article explains in detail how complex web services can be handled with the combination of Web Services Pack and XML Pack. This article explains in detail the process of invoking Complex Web Services involving SSL keystores/trust stores, WSDL 2.0 constructs, SOAP Headers with Username Token Authentication information and Session Management information,Complex Arrays as Input using XML Output Stage and Complex Arrays as Output using XML Input Stage. The article also covers configuring SSL in Tomcat, Debugging SSL Web Services, composing SOAP Headers using SOAPUI and Tcpmon. All of the above features will be explained with the help of sample DataStage jobs and Axis Web Services.

IBM Information Server

IBM InfoSphere Information Server is a unified and comprehensive information integration platform. It profiles, cleanses, and transforms data from heterogeneous data sources to deliver consistent and accurate business data. IBM Information Server is an ideal solution to integrate and synchronize XML data with the business data stored within back end enterprise resource planning (ERP) systems and other data analytical applications.

DataStage ASB Packs

ASB(Application Services Backbone) Packs are DataStage Plugin Components that help in transforming and integrating XML and Web Services data.XML Pack v2.0,Web Services Pack v2.0 and Java Pack v2.0 are collectively known as ASB Packs.

Using the Web Services Pack, you can access web service operations within an IBM InfoSphere Information Server DataStage job. Web Services Pack includes plug-in stages and added functionality in WebSphere DataStage Server routines. In order to learn how DataStage Web Services PACK jobs are designed please follow the developer works article present at http://www.ibm.com/developerworks/data/library/techarticle/dm-0912dominoappsinfosphere

Using XML Pack v2.0,you can perform transformations between XML and relational data.XML Input Stage transforms XML data into relational data.XML Output Stage transforms relational data into XML data.XML Transformer Stage applies a stylsheet to the XML data and converts it into HTML data. In this article we will be using XML Input and XML Output Stages only.

Using Java Pack v2.0,you can call external Java applications developed using the Java Pack API from DataStage.We will not be using Java Pack in this article.

Overview of web services

A Web service is an archive of remote operations that can be called by sending messages over the Internet. A Web service provider publishes a Web service for query and use, and a Web service consumer calls operations from the service. A Web service provider makes available a WSDL (Web Services Description Language) document that defines the service interface. The WSDL document is in XML format. What happens behind the interface is up to the provider, but most providers map the interface to procedure calls in a supported programming language. Incoming requests from a consumer are passed through to the underlying code, and results are passed back to the consumer.

Simple and Complex web services

Simple web services

Simple Web Services are characterized by the following features.

  • Single row as input and single row as output
  • Deployed over HTTP without security
  • Missing Firewall
  • WSDL with Primitive types.

Complex Web Services

Complex Web Services are normally implemented using one or more(but not limited to) of the following features.

  • SSL Authentication
  • Arrays of rows in Input
  • Arrays of rows in Output
  • Arrays of rows in both Input and Output
  • SOAP Headers with Username Token Authentication
  • SOAP Headers with Session ID Information
  • WSDL with Complex types.

DataStage Web Services Pack cannot invoke complex web services out of the box. It needs to be used in combination with XML Pack and other tools like SOAPUI and Tcpmon in order to invoke them.This article explores in detail the various options available in DataStage ASB Packs v2.0 to invoke some of the complex web services listed above.

Pre-requisites

The reader is advised to go through the following articles before trying the scenarios described in the rest of this article.

Please refer the article on creating DataStage Web Service Pack jobs.

Please refer the article for information on creating DataStage XML Pack jobs.

SSL Authentication

Security is a major concern in Web Services Implementation. In an unsecure environment SOAP messages are transferred in plain text over the network, so anyone with a sniffer could intercept the SOAP message and read it. In order to avoid such conditions the web service must be secured with SSL.The SSL and TLS protocols have become the de facto standard for securing network communications. These protocols provide confidentiality, authentication and message integrity. In order to achieve this,the Web Service Container should be SSL enabled by using a Digital Certificate issued by a Certificate Authority. The Web Service Client needs the Certificate details to invoke the Web Service. The SSL and TLS protocols define the rules SSL clients and servers use to communicate with each other. These rules specify the order in which messages are sent, the format of each message, and the way cryptographic algorithms are applied to network communications.

SSL Authentication in DataStage Web Services Pack v2.0

In order to access the Web Service deployed over HTTPS DataStage Web Service Client should fetch the server certificate from the Web Service Container and create a keystore file using keytool and import the server certificate in this keystore. The path of the generated keystore file must be mentioned in the Web Service Client->Stage Tab->Security Tab.Check the SSL Encryption Required> option and mention the path of the keystore file in the Keystore File option.In this sample, SSL is enabled on Tomcat and an Axis Web Service is deployed over it.

Please refer the link http://tomcat.apache.org/tomcat-6.0-doc/ssl-howto.html for details on configuring SSL in Tomcat Server

Please refer the job named WS_SSL in the attachment for SSL functionality

SSL Authentication in Web Service Pack
Web Service Client SSL Options

Debugging SSL Web Service

SSL adds additional complexity to web service client-server communications.Understanding SSL connection problems can sometimes be difficult, especially when it is not clear what messages are actually being sent and received.With the help of the JVM property, javax.net.debug, we can turn on the SSL socket communication debug option. The debug output message can help you to know what exactly happens at the SSL layer. To enable SSL debugging in the Web Service Pack, add the JVM option -Djavax.net.debug in Web Service Client(Grid Style)->Stage Tab->Properties Tab->JVM Options and set it to true.When the DataStage job is compliled and run you can view the SSL debug log messages in the Director Log.

SSL Debugging in Web Service Pack
SSL Debugging
Listing 1. SSL Debug Messages in Director Log.
setting up default SSLSocketFactory
use class specified by ssl.SocketFactory.provider: com.ibm.jsse2.SSLSocketFactoryImpl
class com.ibm.jsse2.SSLSocketFactoryImpl is loaded
init keymanager of type IbmX509
IBMJSSEProvider Build-Level: -20090329
keyStore is: C:\\IBM\\InformationServer\\ASBNode\\apps\\jre\\lib\\security\\cacerts
keyStore type is: jks
keyStore provider is: 
init keystore
trustStore is: C:\\ComplexWebServices\\SSL_WebService\\tomcatkeystore
trustStore type is: jks
trustStore provider is: 
init truststore
adding as trusted cert:
  Subject: CN=santhosh, OU=ibm, O=ibmisl, L=hyd, ST=ap, C=91
  Issuer:  CN=santhosh, OU=ibm, O=ibmisl, L=hyd, ST=ap, C=91
  Algorithm: RSA; Serial number: 0x4bd97c41
  Valid from Thu Apr 29 18:02:01 IST 2010 until Wed Jul 28 18:02:01 IST 2010

The above SSL debug messages show important information like the type of SSLSocketFactory,type of keystore/truststore,Algorithm used for encryption(RSA) and validity of the certificate etc.

Please refer the job named WS_SSL_Dbg for SSL Debugging options in Web Service Pack

Web services with Complex SOAP Headers

SOAP Headers

SOAP(Simple Object Access Protocol) is an XML based communication protocol used by web services. Web Services communication is performed in terms of SOAP Messages. An application(normally a web service client) invokes a web service by sending it a SOAP Request and the web service responds to the application by sending it a SOAP Response. The SOAP Request which is an XML document consists of a root Envelope element. The Envelope elements consists of a mandatory Body element and an optional Header Element. The Body element contains information intended for the web service to process and the Header Element contains application specific information like authetication data,session data etc.

SOAP Header with Secure Username Token Information

Composing Username Token Information in SOAP Header

Apache Rampart is the Axis2 module that implements the WS-Security functionality based on Apache WSS4J. WS-Security provides multiple ways in which one can authenticate a user when they need to access a service. One easy mechanisms is to use a UsernameToken. In Username Token Authentication the user credentials are placed within the SOAP Header.The purpose of a UsernameToken is just to convey username and password information as part of the WS-Security headers. The most basic form of UsernameToken sends both the username and password as plain text.

Web Service Pack v2.0 by itself cannot compose SOAP Header information. The user has to compose the SOAP Header before hand in order to access the Web Service.One such method of composing the SOAP header is by invoking the Web Service from a third party web service client,capture the SOAP Request using TCPMON and fetch the SOAP Header from it. Use the Header thus obtained in the Web Service Client Stage to invoke the Web Service. I have used Axis2 UsernameToken Sample Services provided at the URL http://wso2.org/library/240. Follow the link to generate the web service and deploying it.You can run the sample Axis2 Client provided to access the web service. In order to capture the SOAP Request use TCPMON.Please follow the link http://ws.apache.org/commons/tcpmon/tcpmontutorial.html to know about TCPMON and its usage. Following is the screenshot of the TCPMON with SOAP Request and Response.

TCPMON with SOAP Request and Response
TCPMON with SOAP Request and Response

Invoking Username Token Web Service from Web Services Pack

The above step captured the Header Information with Username Token.The Web Services Pack has a provision to mention the SOAP Header explicitly in the Stage itself. Switch to Web Service Client > Output Tab > Input Arguments and insert a new request value named SOAPHEADER. In the value field mention/copy the SOAP Header information and check the Header checkbox. Compile and Run the DataStage job. The Web Service Client will call the Web Service by composing the complete SOAP Request from the user defined SOAP Header in the Web Service Stage Properties.

Web Service Client Stage:SOAP Header Option
Web Service Client Stage:SOAP Header Option
Web Service Client Stage:SOAP Header Option
Web Service Client Stage:SOAP Header Option

Following is the complete SOAP Request generated by DataStage Web Services Pack

Listing 2. SOAP Request with Username Token information in the SOAP Header
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Header>
<wsse:Security soapenv:mustUnderstand="1" 
xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-
            wss-wssecurity-secext-1.0.xsd">
<wsse:UsernameToken wsu:Id="UsernameToken-63046594" 
xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-
            wss-wssecurity-utility-1.0.xsd">
<wsse:Username>bob</wsse:Username>
<wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-
        username-token-profile-1.0#PasswordDigest">rW9ziD+5CnOrXZ4kxErEMJvs9y8=
            </wsse:Password>
<wsse:Nonce>nATeerIJqyHxHEOyKsRFvg==</wsse:Nonce>
<wsu:Created>2010-02-05T12:24:59.718Z</wsu:Created>
</wsse:UsernameToken>
</wsse:Security> 
</soapenv:Header>
<soapenv:Body><ns1:echo 
xmlns:ns1="http://sample02.samples.rampart.apache.org" 
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ns1:param0>THIS IS A SAMPLE USERNAME TOKEN SERVICE</ns1:param0></ns1:echo> 
</soapenv:Body>
</soapenv:Envelope>

Please refer the job named WS_UserNameToken in the attachment for this scenario.

SOAP Header with user Session Information

Consider the following Web Service invocation scenario.

A first invocation of the web service with username/password would return a user session id.

The second web service invocation must be done by composing SOAP Header using the Session ID obtained from the response of the first invocation of Web Service.

It is the second web service response that contains the actual information returned by the web service.This is a common scenario with CRM and SaaS based web services like SalesForce.com

The first call to the web service is done normally without any headers.In order to make the second call using the Session ID from the first call we need a third party web service client called soapUI.Using soapUI we need to figure out the composition and hierarchy of the SOAP Header that needs to be used in the Web Service Client Stage.

Finding the SOAP Header using soapUI

Launch the soapUI executable.Create a new soapUI Project by clicking on File->New soapUI Project. Provide a Project name(SessionProject in my case),browse for the WSDL file of the Web Service and click on ok.

soapUI Project
soapUI Project

soapUI will load the WSDL definitions and displays an hierarchical view of the supported web service operations in the left pane.

soapUI Web Service Operations
soapUI Web Service Operations

In order to check the structure of the SOAP Request, in particular the SOAP Header click on the Request option under the operation name in the hierarchical view. The SOAP Request containing the SOAP Header will be displayed in the right pane.

soapUI SOAP Request with Header
soapUI SOAP Request with Header

Extract this SOAP Header and mention it as part of the Header Element in the Web Service Client Stage in the same way as mentioned for the UserName Token Scenario. Compile and Run the Web Service Client job.

Please refer the jobs named sf_login,sf_columns and sf_xml in the attachment for this scenario.

Arrays of rows as Input to the Web Service

Arrays of rows as Input to the Web Service:Normal Operation

The Web Services Pack sends requests, one row at a time to the web service, which are encoded in a SOAP message. In the simplest case, a web service operation expects a single parameter value as input to an operation, and your input data contains rows with a single value. Each row is processed in a separate request. Sending each row is a seperate SOAP Request results in significant Performance degradation since it involves the following operations.

  • Compose SOAP Request for each input row
  • Call the Web Service
  • Fetch the SOAP Response
  • Parse the SOAP Response XML and retrive the Response Value

Consider a Web Service that simply consumes a row and does some processing on it. If this web service is called with five rows from Web Services Pack, five corresponding SOAP Requests will be created and five Web Service calls will be made. Following is the DataStage Job and corresponding Director Log. We can observe that there are five corresponding SOAP Requests and Responses.

Web Services Stage:Arrays of rows in Input
Web Services Stage:Arrays of rows in Input
Listing 3.Director Log with SOAP Message Logging
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0029':'
[WebService] ---- XML request envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0030':'
[WebService] ---- XML response envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0029':'
[WebService] ---- XML request envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0030':'
[WebService] ---- XML response envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0029':'
[WebService] ---- XML request envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0030':'
[WebService] ---- XML response envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0029':'
[WebService] ---- XML request envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0030':'
[WebService] ---- XML response envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0029':'
[WebService] ---- XML request envelope ---- (...)
ConsumeRows,0: Info: WSPACK_InputRowsWithoutXMLOutput.ConsumeRows: 0030':'
[WebService] ---- XML response envelope ---- (...)
Listing 4. SOAP Request created for each individual row
FOR THE FIRST ROW
#########################################################
<soapenv:Envelope 
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<ns1:ConsumeRows 
xmlns:ns1="http://RowConsumer.MultipleRowsAsInput.isd.ibm.com/soapoverhttp/"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<arg1>
<RowConsumerInVar1><field001>row1</field001><field002>the first sentence</field002>
</RowConsumerInVar1>
</arg1>
</ns1:ConsumeRows>
</soapenv:Body>
</soapenv:Envelope>

FOR THE SECOND ROW
#########################################################
<soapenv:Envelope 
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<ns1:ConsumeRows 
xmlns:ns1="http://RowConsumer.MultipleRowsAsInput.isd.ibm.com/soapoverhttp/"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<arg1>
<RowConsumerInVar1><field001>row2</field001><field002>the second sentence</field002>
</RowConsumerInVar1>
</arg1>
</ns1:ConsumeRows>
</soapenv:Body>
</soapenv:Envelope>

**********************AND SO ON FOR THE REMAINING THREE ROWS

Please refer the job named WS_ArrarInv1 in the attachment for this scenario.

The following section would address the above problem by introducing the XML Output Stage to compose the SOAP Body

Arrays of Rows as Input to the Web Service:Improved using XML Output Stage

XML Output Stage is a DataStage Plugin that is part of XML Pack 2.0. The XML Output Stage generates XML data from relational data. The idea here is to use XML Output Stage to compose the SOAP Body by aggregating the five rows from the flat file, pass this SOAP Body to the Web Service Stage so that the complete SOAP Request is created. Following is the corresponding job design.

Arrays of Rows as Input with XML Output Stage
Arrays of Rows as Input with XML Output Stage

In order to compose the SOAP Body,pass the flat file data to XML Output Stage.Populate the XML Output Stage properties using the web service definitions imported from the WSDL file of the corresponding web service.The imported web service defintions are available in the DataStage Designer at Repository > DataStage Project Name > Table Definitions > Web Services > Web Service Name(RowConsumer in my case). The table definition that must be used to populate the XML Output Stage will be of the form OperationName_IN(ConsumeRows_IN in this case).

WSDL Definitions
WSDL Definitions

The Peek Stage is used to have a look at the incoming SOAP Body XML from the XML Output Stage in the Director Log.It is not mandatory.The SOAP Body will be part of the column defined in the XML Output Stage > Output Tab > Columns Tab(SoapBody in my case).

SOAP Body column in the Output Tab of XML Output Stage
SOAP Body column in the Output Tab of XML Output Stage

The SOAP Body is fed to the WS Client Stage through the Output Link of Peek Stage. Load the Stage Properties of the WS Client using the standard procedure by importing the web service definitions using the WSDL provided.In the WS Client Stage,Input Tab > Input Message Tab check the option named user defined message and select the column holding the SOAP Body (SOAPBody in this case).

Column holding the SOAP Body
Column holding the SOAP Body

Compile and run the job.In the Director Log you can observe that only one SOAP Request is sent by the WS Client Stage instead of five. The XML Output Stage has aggregated the five rows in the SOAP Body and passed on the same to WS Client. The WS Client has used the SOAP Body to create the complete SOAP Request and invoke the Web Service. In this way extra overhead of creating the SOAP Request and calling the web service for each individual row has been eliminated.

Listing 5. SOAP Body generated by XML Output Stage
<?xmlversion="1.0"encoding="UTF-8"?>
<ns1:ConsumeRowsxmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:ns1="http://RowConsumer.MultipleRowsAsInput.isd.ibm.com/soapoverhttp/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<arg1>
<RowConsumerInVar1>
<field001>row1</field001><field002>thefirstsentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row2</field001><field002>thesecondsentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row3</field001><field002>thethirdentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row4</field001><field002>thefourthsentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row5</field001><field002>thefifthsentence</field002>
</RowConsumerInVar1>
</arg1>
</ns1:ConsumeRows>

Here is the complete SOAP Request generated by WS Client Stage

Listing 6. SOAP Request generated by WS Client Stage
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body><ns1:ConsumeRows
xmlns:ns1="http://RowConsumer.MultipleRowsAsInput.isd.ibm.com/soapoverhttp/" 
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<arg1>
<RowConsumerInVar1>
<field001>row1</field001>
<field002>thefirstsentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row2</field001>
<field002>thesecondsentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row3</field001>
<field002>thethirdsentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row4</field001>
<field002>thefourthsentence</field002>
</RowConsumerInVar1>
<RowConsumerInVar1>
<field001>row5</field001>
<field002>thefifthsentence</field002>
</RowConsumerInVar1>
</arg1>
</ns1:ConsumeRows> 
</soapenv:Body>

Please refer the job named WS_ArrayInv2 in the attachment for this scenario.

Handling Arrays of rows as Output from the Web Service

Arrays of rows as output from the Web Service:Normal Operation

The Web Service Pack cannot handle multiple arrays as output. It can only fetch one row at a time from the web service response. A web service returns a SOAP message that the Web Services Client stage can decode into columns of one output row using XPath expressions. If you need to decode the SOAP message into multiple rows,the XML Input stage must be used

Consider a Web Service that sends data of five employees as output in its response.In the normal operation the web service pack will only fetch a single row as output

Arrays of Rows As Output
Arrays of Rows As Output

Please refer the job named WS_ArrayOutv1 in the attachment for this scenario.

Arrays of rows from the Web Service:Improved using XML Input Stage

XML Input Stage is a DataStage Plugin that is part of the XML Pack v2.0. It parses XML data and generates relational data. The idea here is to fetch the SOAP Body XML from the Web Service Client instead of the flattened data and pass this XML to XML Input Stage to flatten it into rows and columns.

With XML Input Stage
With XML Input Stage

The SOAP Body is sent to XML Input Stage without any decoding. To do so, define a column in the input link of the receiving stage(Peek Stage above). On the Output tab of the Web Services Client stage, click the Output Message tab. Select the User-Defined Message check box. In Choose the Column Receiving the User Message drop down list, select the column defined in the linked stage that will receive the output message.

Column holding the SOAP Response Body
Column holding the SOAP Response Body
Column Receiving the SOAP Response Body
Column Receiving the SOAP Response Body

Pass this column containing the SOAP Body to XML Input Stage through the Peek Stage(not mandatory. You can pass it directly without peek stage as well). In order to parse this SOAP Body XML in XML Input Stage we need to populate the XML Input Stage with the SOAP Body XML table definitions. To do so,use the table definition named operation_OUT present in Repository > Project Name > Table Definitions > Web Services > Web Service Name. In this case it is EmpData_OUT

WSDL Definitions
WSDL Definitions

Finish the job design by linking the XML Input Stage with a Sequential File Stage so that the flattened SOAP Body data is captured in a text file. Save the job,compile and run it. You can observe that the WS Client Stage has captured the SOAP Body and passed the same to the XML Input Stage for transforming it into rows and columns.

Listing 7. SOAP Response Body XML captured by WS Client Stage
<?xml version="1.0" encoding="UTF-8"?>
<p725:EmpDataResponse xmlns:p725="http://EDataSer.EmpData.isd.ibm.com/soapoverhttp/">
<EmpDataReturn>
<EDataSerOutVar1>
<empid>1123</empid><empname>John Millers</empname>
<empage>24</empage><empdept>Sales</empdept>
<emploc>Los Angeles</emploc>
</EDataSerOutVar1>
<EDataSerOutVar1>
<empid>1133</empid><empname>Mike Gordon</empname>
<empage>26</empage><empdept>Production</empdept>
<emploc>Chicago</emploc>
</EDataSerOutVar1>
<EDataSerOutVar1>
<empid>1143</empid><empname>Ruez Xavier</empname>
<empage>28</empage><empdept>Quality Assurance</empdept>
<emploc>Las Vegas</emploc>
</EDataSerOutVar1>
<EDataSerOutVar1>
<empid>1153</empid><empname>Gary Sobers</empname>
<empage>30</empage>
<empdept>Finance</empdept>
<emploc>Hong Kong</emploc>
</EDataSerOutVar1>
<EDataSerOutVar1>
<empid>1163</empid><empname>Ian Smith</empname>
<empage>32</empage>
<empdept>Documentation</empdept>
<emploc>Singapore</emploc>
</EDataSerOutVar1>
</EmpDataReturn>
</p725:EmpDataResponse>
Listing 8. Flattened Data produced by XML Input Stage
"1123","John Millers","24","Sales","Los Angeles"
"1133","Mike Gordon","26","Production","Chicago"
"1143","Ruez Xavier","28","Quality Assurance","Las Vegas"
"1153","Gary Sobers","30","Finance","Hong Kong"
"1163","Ian Smith","32","Documentation","Singapore"

Please refer the job named WS_ArrayOutv2in the attachment for this scenario.

Arrays of Rows in both Input and Output

Arrays of Rows in Input and Output: Using both XML Input Stage and XML Output Stage

In this scenario we need to use a combination of the above two scenarios. Use XML Output Stage to compose the SOAP Request Body by aggregating the input rows from the flat file data to be sent in the Web Service SOAP Request and use XML Input Stage to parse the SOAP Response Body XML and flatten it into rows and columns.

With XML Input and XML Output Stage
With XML Input and XML Output Stage
Listing 9.SOAP Request Body composed by XML Output Stage
<soapenv:Body>
<ns1:UpperCase xmlns:ns1="http://CaseConv.MultipleRowsInOut.isd.ibm.com/soapoverhttp/" 
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<arg1>
<CaseCoversionInVar1>
<field001>row1</field001>
<field002>the first sentence</field002>
</CaseCoversionInVar1>
<CaseCoversionInVar1>
<field001>row2</field001>
<field002>the second sentence</field002>
</CaseCoversionInVar1>
<CaseCoversionInVar1>
<field001>row3</field001>
<field002>the third sentence</field002>
</CaseCoversionInVar1>
<CaseCoversionInVar1>
<field001>row4</field001>
<field002>the fourth sentence</field002>
</CaseCoversionInVar1>
<CaseCoversionInVar1>
<field001>row5</field001>
<field002>the fifth sentence</field002>
</CaseCoversionInVar1>
</arg1>
</ns1:UpperCase> 
</soapenv:Body>
Listing 10. SOAP Response Body captured by XML Input Stage
<soapenv:Body>
<p266:UpperCaseResponse xmlns:p266="http://CaseConv.
            MultipleRowsInOut.isd.ibm.com/soapoverhttp/">
<UpperCaseReturn>
<CaseCoversionOutVar1>
<field001>ROW1</field001>
<field002>THE FIRST SENTENCE</field002>
</CaseCoversionOutVar1>
<CaseCoversionOutVar1>
<field001>ROW2</field001>
<field002>THE SECOND SENTENCE</field002>
</CaseCoversionOutVar1>
<CaseCoversionOutVar1>
<field001>ROW3</field001>
<field002>THE THIRD SENTENCE</field002>
</CaseCoversionOutVar1>
<CaseCoversionOutVar1>
<field001>ROW4</field001>
<field002>THE FOURTH SENTENCE</field002>
</CaseCoversionOutVar1>
<CaseCoversionOutVar1>
<field001>ROW5</field001>
<field002>THE FIFTH SENTENCE</field002>
</CaseCoversionOutVar1>
</UpperCaseReturn>
</p266:UpperCaseResponse>
</soapenv:Body>

Please refer the jobs named WS_ArrayInOutv1 and WS_ArrayInOutv2 in the attachment for this scenario.

The article started with an introduction to IBM InfoSphere Information Server, DataStage ASB Packs,simple and complex web services. The later sections explained in detail how to invoke complex web services deployed over SSL, web services which require Header information in the SOAP Requests, web services which accept Arrays as input, web services which return Arrays in output and a combination of both. All the scenarios are explained using DataStage jobs and corresponding screenshots are provided as well.


Download

DescriptionNameSize
DataStage Jobs for the articleArticleJobs.zip32KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into SOA and web services on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=SOA and web services
ArticleID=649015
ArticleTitle=Handling Complex Web Services in InfoSphere Information Server through DataStage ASB Packs v2.0
publish-date=05242011