Use DataStage Web Services Pack to fetch Lotus Notes data into an InfoSphere Information Servers DataStage module for more ETL processing

A step-by-step guide

This article explores various features that Lotus® Domino® Server provides to create and deploy Lotus applications as Web services. You will learn how IBM InfoSphere® Information Server uses these applications through DataStage® Web Services Pack. This step-by-step guide helps you create, configure, and deploy Web services in Lotus Domino, and it helps you create, configure, compile, and execute a DataStage Web Services Pack job that uses Lotus Web services.

Santhosh Kumar Kotte (sankotte@in.ibm.com), Systems Software Engineer, IBM  

Photo of Santhosh KotteSanthosh Kotte is a Systems Software Engineer working for IBM InfoSphere Information Server Group at the India Software Lab, Hyderabad. He joined IBM as an Associate Software Engineer in 2007. He works on the development and maintenance of DataStage ASB Packs (XML Pack 2.0, Java Pack 2.0, and Web Services Pack 2.0) and on the next generation Web Services Pack (WS Pack 3.0). His primary interests are working with emerging technologies, such as XML, DB2 pureXML, SOA, and Web services, and exploring various information integration technologies.



Kartika Garg (kartgarg@in.ibm.com), Advisory Software Engineer, IBM  

Photo of Kartika GargKartika Garg is part of the IBM InfoSphere Information Server group based in the India Software Laboratory, Hyderabad. She joined IBM in 2006 as an experienced professional. She has been working in the areas of ETL tools, data integration, performance improvement, and telecommunication systems for 10 years. Currently, as a connectivity developer, she is involved in development and maintenance of ODBC and Sybase connectivity for Information Server.



Ritesh Kumar Gupta (ritesh.kumar.gupta@in.ibm.com), Architect, IBM  

Photo of Ritesh GuptaRitesh Kumar Gupta is part of the IBM InfoSphere Information Server group based in the India Software Laboratory, Hyderabad. He joined IBM in 2005 as part of the Ascential acquisition. He has been working in the areas of ETL tools, data integration, performance improvement, and telecommunication systems for 13 years. Currently, as a connectivity architect, he is involved in various integration activities within InfoSphere products in additiona to exploring various vendor-specific integration solutions for IBM InfoSphere Information Server.



10 December 2009

Also available in Chinese

Introduction

InfoSphere® Information Server's DataStage® module provides many operators and plug-ins for various data sources such as Oracle, DB2®, XML, Teradata, and SAP. But the DataStage module does not include a dedicated plug-in or operator to fetch Lotus Notes® data into DataStage to perform ETL operations.

Many customers have asked for this function. The common requirement is to fetch Notes data into DataStage, perform an extract, transform, and load (ETL), and then pass this data to other Information Server modules for more ETL processing. You cannot use the Lotus SQL driver, because it has issues on UNIX®-based systems due to incompatibilities with the latest ODBC standards. You cannot use third-party ODBC-to-ODBC bridges, because they are licensed. A cost-effective and novel way to fetch Lotus Notes data into InfoSphere Information Servers DataStage module for more ETL processing is needed.

This article explores various features that Lotus Domino® Server provides to create and deploy Lotus applications as Web services. You will learn how IBM InfoSphere Information Server uses these applications through DataStage Web Services Pack. This step-by-step guide helps you create, configure, and deploy Web services in Lotus Domino, and it helps you create, configure, compile, and execute a DataStage Web Services Pack job that uses Lotus Web services.

Introducing the products

This section describes the IBM Information Server, DataStage Web Services Pack, Lotus Domino, and the Lotus Domino suite of products.

IBM Information Server and the DataStage Web Services Pack

IBM InfoSphere Information Server is a unified and comprehensive information integration platform. It profiles, cleanses, and transforms data from heterogeneous data sources to deliver consistent and accurate business data. IBM Information Server is an ideal solution to integrate and synchronize XML data with the business data stored within back-end enterprise resource planning (ERP) systems and other data analytical applications.

Using the DataStage Web Services Pack, you can access Web service operations within an IBM InfoSphere Information Server DataStage job. The Web Services Pack includes plug-in stages and added function for the DataStage server routines.

Lotus Domino and its suite of products

As integral parts of the IBM Workplace family, IBM Lotus Notes and Domino software enable users to share, manage, and organize information more efficiently. Lotus Notes and Domino 7 software products deliver a reliable, security-rich messaging and collaborative environment that can help companies enhance people's productivity, streamline business processes, and improve overall business responsiveness. Lotus Domino is an application server that provides groupware functions and development tools to create messaging, collaboration, workflow, tracking, Internet, and intranet applications. The Domino server stores information in Lotus Notes databases. All information, including POP3, HTTP, and Web services, resides in these Notes databases, which are the building blocks developers and administrators use to create applications and services. Notes databases store information and keep the extra function required to develop applications on the Lotus Domino platform.

IBM Lotus Domino Designer offers application development software that enables application developers and Web site designers to create security-rich, collaborative applications that can be accessed through Lotus Notes or supported Web browsers. Developers can easily create core business solutions based on several factors, including:

  • Forms
  • Views
  • Pages
  • Framesets
  • Integrated instant messaging
  • XML
  • Java™
  • JavaScript

Introducing Lotus Domino Web services

This section describes Lotus Domino and Web services.

Overview of Web services

A Web service is an archive of remote operations that can be called by sending messages over the Internet. A Web service provider publishes a Web service for query and use, and a Web service consumer calls operations from the service. A Web service provider makes available a Web Services Description Language (WSDL) document that defines the service interface. The WSDL document is in XML format. What happens behind the interface is up to the provider, but commonly providers map the interface to procedure calls in a supported programming language. Incoming requests from a consumer are passed through to the underlying code, and results are passed back to the consumer.

Lotus Domino and Web services

Lotus Domino V7.0 introduced the new Web service design element in Lotus Domino Designer. If you open a database in the Lotus Domino Designer V7.0 client, you see the Web Services entry just below the familiar Agents entry in the Shared Code section of the design element tree. Lotus Domino maps the WSDL interface to an agent-like Web service design element that can be coded in Lotus Script or Java. To be used, the Web service must be on a Domino server with HTTP enabled.

Lotus Domino handles all the WSDL creation and SOAP handling for you, so all you have to do is to write code in your Web service design element as though you were coding an agent. As long as you specify which class to use as the interface class for the service, Lotus Domino can publish a WSDL file, can convert incoming SOAP requests to method calls on your class, and can return the results of your method (if any) as a SOAP response. From a coding standpoint, all you do is write a Lotus Script or Java class. Lotus Domino does the rest


Creating a Lotus Domino Web Service

You can use any of several approaches to create a Web service design element in Domino Designer. If you code it entirely in Lotus Script or in Java, saving the design element generates a WSDL document that reflects the Lotus Script or Java code. If you import an existing WSDL document, Lotus Script or Java is generated that reflects the operations in the imported WSDL. The Web service design element saves the WSDL document and the code. If the public interface has not changed, the WSDL document stays as is. If, in your coding, you change anything that affects the public interface, a new WSDL is generated. In Domino Designer, the Web service design element resides below Agents under Shared Code.

  1. Ensure that in Domino Administrator, the Programmability Restrictions fields under Server document > Security tab have the names of the people who sign the Web services, including the administrator and the server. By default, these fields are blank, allowing no access.
  2. Launch Domino Designer.
  3. Create a new blank database by clicking File > Database > New or CTRL+N, or open an existing database by clicking File > Database > Open or CTRL+O. In the design view, you see that a new design element called Web Services is added to a Lotus Notes database, as shown in Figure 1.
Figure 1. Domino Designer and its Web service element
Screen cap: Design - Web Services window in Domino Designer showing Webservices tab selected
  1. Click New Web Service to generate a new Web service.

Coding the new Web service

  1. Code a class within this Web service design element using either Lotus Script or Java. All the methods of the class are automatically exposed as Web service methods.
  2. If you do not want a particular method to be exposed, change it to a Private Method by adding the Private qualifier before the function.

Basic coding

The code for a Web service has the following elements:

  • A class definition for the implementation code: this class must become the PortType class named in the Basics tab of the properties box, and the class must be public.
  • Within the class, a procedure (function, sub, or method) definition for each operation in the Web service: these procedures must be public. Supporting procedures that you don't want in the interface must be private.
  • Inclusion of Lotus Script .lss files or importing of java packages
  • Initialization of a Notes Session (LotusScript) or Session (Java) objects if Domino Objects are accessed.

Listing 1 and Figure 2 show how to create a very simple Lotus Script class that uses only one method to add two numbers and to return the result.

Listing 1. Sample Lotus Script code that adds two numbers
Class MathService
Function Sum(X as Integer, Y as Integer) As Integer
Sum = X+Y
End Function
End Class
Figure 2. Lotus Script to add two numbers
Screen cap: MathService - Web Service tab with Listing 1 content

Setting Web Service properties

The full documentation to set properties is available in the Domino Designer documentation Resources, so this article highlights only some of the important parts.

  1. Right-click the coding pane, and click Web Service Properties to bring up the properties dialog.
  2. Set the name of the Web Service Design element, such as Math.
  3. Specify which class is to be exposed as a Web service, such as MathService.
  4. Provide a name (required).
  5. Optionally provide an alias or comment.
  6. Optionally specify that you want to be warned if a coding change causes the generation of a new WSDL.
  7. Provide the PortType class, which is the name of the class that defines the procedures that map to the WSDL operations. These procedures must be public functions or subs in Lotus Script, or they must be public methods in Java. Private functions, subs, and methods are not exposed through the Web services interface. Note that you cannot enter the PortType class in the Properties window until you have created the class by coding or by importing a WSDL, as shown in Figure 3.
Figure 3. Web Service properties
Screen cap: MathService - Web Service tab with Web Service Properties window showing Name = MathService and PortType=MathService
  1. In the Security tab, specify that you want the Web service to run with the effective user name of the user invoking the Web service by checking Run as web user. For the simple Math example, you don't need to run any restricted operations, as shown in Figure 4.
Figure 4. Web Service properties
Screen cap: Web Service properties window showing Run as Web User, Allow Remote Debugging, Profile This Web Service and All Readers and Above checked

The Advanced tab enables you to fine tune the Soap Message parameters to suit your needs, as shown in Figure 5.

Figure 5. SOAP message parameters on Advanced tab
Screen cap: Web Service properties window Advanced tab showing Programming Model = RPC, SOAP Message Format = RPC/encoded, Include Op name in SOAP=checked, Port Type Name=MathService, Service Element Name=MathServiceService, and Service Port Name=Domino

Accessing the Web service

The following are example URLs to access the Web service:

  • The WSDL definition for the Web service that you created is available at http://servername<:port>/databasename.nsf/webservicename?WSDL
  • In the example, this translates to http://9.182.77.93/webservi.nsf/MathService?WSDL
  • The Web service itself is available at http://servername<:port>/databasename.nsf/webservicename?OpenWebService
  • In the example, this translates to http://9.182.77.93/webservi.nsf/MathService?OpenWebService

Accessing the Lotus Domino Web service through Web Service Pack

High-level design architecture

This section describes how InfoSphere Information Server deployed on IBM WebSphere Application Server communicates with a Web service application deployed on Lotus Domino Server. The DataStage Web Services Pack constructs a SOAP request from the user-defined inputs and sends it over HTTP. The Domino application receives the SOAP request with the exposed Web service (the .nsf file that contains the Web service design element). The Domino Web service application processes the SOAP request and the result is passed to the DataStage Web Service Pack in the form of a SOAP response over HTTP. The Web Services Pack in turn parses the SOAP response, fetches the response value, and passes it to the DataStage user. For the example, the response value is logged to the DataStage Director. Figure 6 illustrates this architecture flow.

Figure 6. DataStage and Domino integration
Diagram showing (A) Requests and Responses going between Lotus Domino server tasks and the database with Web service design element. (B) DataStage Web service pack and Quality stage running any ETL tool. Then Requests and Responses going between A and B.

Importing the WSDL metadata

You need to import message and metadata information from the WSDL document for each Web service operation that you want to call from a Web Services Pack stage. WSDL documents are parsed and converted to table definitions. To access and import WSDL documents, use the Web Service Metadata Importer.

  1. Open the InfoSphere DataStage Designer.
  2. Select Import > Table Definitions > Web Service Table Definitions, as shown in Figure 7. The Web Service Metadata Importer opens.
Figure 7. DataStage Designer
Screen cap: DataStage Designer showing selected menu options
  1. Enter the URL for the WSDL document in the Address field. For the example, enter http://servername<:port>/databasename.nsf/webservicename?WSDL, as shown in Figure 8.
Figure 8. WSDL importer
Screen cap: Web Service Meta Data Importer window showing C:\Documents and Settings\Administrator\HYDCONDEVWIN01\Desktop\LotusPics\MathService.wsdl in the Address field
  1. The Web Service Explorer window displays a tree view of several WSDL elements. Highlight one or more operations in the Web Service Explorer window.
  2. Right-click your selections, and select Import. The Import Progress window opens, as shown in Figure 9.
Figure 9. WSDL Importer: Import Progress
Screen cap: Import Progress window
  1. Click Details to see the list of imported messages.
  2. Click Close to complete the import.

Setting up the Web Service Pack stage and the DataStage job

  1. From the Real Time category of the Palette window, drag the Web services client stage icon onto the canvas.
  2. Connect other stages based on whether the Web service acts as a data source or as a data target.
  3. Attach a target stage to the Web service stage to which the response of the Web service will be written. For the example, use a Peek Stage, which prints the Web service response in the director log, as shown in Figure 10.
Figure 10. DataStage parallel job
Drawing of Access_Lotus_DB linking using DSLink2 to View_Result
  1. In the Web Services client stage window, click the General tab.
  2. Click Select Web Service Operation, as shown in Figure 11. The Web Service browser window opens.
Figure 11. Web service client stage
Screen cap: Access_Lotus_DB - WSClientPX stage window showing Select Web Service Operation button to select
  1. In the Web services window, highlight the service name.
  2. In the Operations window, highlight the operation that you want to use in the Web services client stage.
  3. Click the Select this item link in the Information window. The Web services client stage window reappears. The Web service name and the selected operation name appear on the General window, as shown in Figure 12.
Figure 12. Web service operation name
Screen cap: Web Service Browser window with MathServiceService selected
Figure 13. Web Service properties
Screen cap: Access_Lotus_DB - WSClientPX stage window showing the Stage tab selected and Web Service information completed
  1. Define the credentials needed to access the Web service using the Security Tab, as shown in Figure 14.
Figure 14. Single sign-on details
Screen cap: Access_Lotus_DB - WSClientPX stage window with Security tab selected and Authentication Required checked
  1. On the Output Tab, click Input Arguments > Load Arguments, as shown in Figure 15. The input arguments to Web service are automatically loaded. A user enters the input values to those arguments.
Figure 15. Web service input arguments
Screen cap: Access_Lotus_DB - WSClientPX stage window with Output tab selected, Output Name=DSLink2 and Input Arguments tab is selected
  1. On the Output Message Tab, click Load Message Information, as shown in Figure 16. All the WSDL namespaces will be populated automatically.
Figure 16. Output message properties
Screen cap: Access_Lotus_DB - WSClientPX stage window with Output Message tab selected and Load Message Information button highlighted
  1. Go to the Columns tab to find the Web service response argument populated automatically, as shown in Figure 17.
Figure 17. Web service output arguments
Screen cap: Access_Lotus_DB - WSClientPX stage window with Output tab selected and Columns tabs selected
  1. Attach a target stage to the Web service stage to which the response of the Web service will be written. For the example, attach Peek Stage, which prints the Web service response in the director log.

Compiling and running the DataStage job

  1. In the DataStage Designer, click the Compile button. A window pops up to show the successful compilation of the job, as shown in Figure 18.
Figure 18. Compile job
Screen cap: Compile Job - AccessDomino window showing the job compiled successfully
  1. Close the window, and click the Run button, as shown in Figure 19. The job runs successfully.
Figure 19. Run the job
Screen cap: AccessDomino - Job Run Options window with Run button selected
  1. Open the DataStage Director, and check the job log. For the example, the Web service returns the sum of 300 in the director log, as shown in Figure 20.
Figure 20. Result in director log
Screen cap: Event Detail window showing View_Result=0; SumReturn=300 in the Message field

Conclusion

The article started with an introduction to IBM InfoSphere Information Server, DataStage Web Services Pack, Lotus Domino, and Lotus Domino Web Services support. You learned how to create Web services using the Web Service Design Element in Domino Designer and how to deploy the Web services in Lotus Domino Server. Later you explored in detail how to configure a Web Service Client Stage in a DataStage Job to invoke the Lotus Domino Web Service.


Download

DescriptionNameSize
Sample DataStage job for this articleDomino_IS.tar30KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Information management on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Information Management, Lotus, SOA and web services
ArticleID=455225
ArticleTitle=Use DataStage Web Services Pack to fetch Lotus Notes data into an InfoSphere Information Servers DataStage module for more ETL processing
publish-date=12102009