Contents


Use DataStage Web Services Pack to fetch Lotus Notes data into an InfoSphere Information Servers DataStage module for more ETL processing

A step-by-step guide

Comments

InfoSphere® Information Server's DataStage® module provides many operators and plug-ins for various data sources such as Oracle, DB2®, XML, Teradata, and SAP. But the DataStage module does not include a dedicated plug-in or operator to fetch Lotus Notes® data into DataStage to perform ETL operations.

Many customers have asked for this function. The common requirement is to fetch Notes data into DataStage, perform an extract, transform, and load (ETL), and then pass this data to other Information Server modules for more ETL processing. You cannot use the Lotus SQL driver, because it has issues on UNIX®-based systems due to incompatibilities with the latest ODBC standards. You cannot use third-party ODBC-to-ODBC bridges, because they are licensed. A cost-effective and novel way to fetch Lotus Notes data into InfoSphere Information Servers DataStage module for more ETL processing is needed.

This article explores various features that Lotus Domino® Server provides to create and deploy Lotus applications as Web services. You will learn how IBM InfoSphere Information Server uses these applications through DataStage Web Services Pack. This step-by-step guide helps you create, configure, and deploy Web services in Lotus Domino, and it helps you create, configure, compile, and execute a DataStage Web Services Pack job that uses Lotus Web services.

Introducing the products

This section describes the IBM Information Server, DataStage Web Services Pack, Lotus Domino, and the Lotus Domino suite of products.

IBM Information Server and the DataStage Web Services Pack

IBM InfoSphere Information Server is a unified and comprehensive information integration platform. It profiles, cleanses, and transforms data from heterogeneous data sources to deliver consistent and accurate business data. IBM Information Server is an ideal solution to integrate and synchronize XML data with the business data stored within back-end enterprise resource planning (ERP) systems and other data analytical applications.

Using the DataStage Web Services Pack, you can access Web service operations within an IBM InfoSphere Information Server DataStage job. The Web Services Pack includes plug-in stages and added function for the DataStage server routines.

Lotus Domino and its suite of products

As integral parts of the IBM Workplace family, IBM Lotus Notes and Domino software enable users to share, manage, and organize information more efficiently. Lotus Notes and Domino 7 software products deliver a reliable, security-rich messaging and collaborative environment that can help companies enhance people's productivity, streamline business processes, and improve overall business responsiveness. Lotus Domino is an application server that provides groupware functions and development tools to create messaging, collaboration, workflow, tracking, Internet, and intranet applications. The Domino server stores information in Lotus Notes databases. All information, including POP3, HTTP, and Web services, resides in these Notes databases, which are the building blocks developers and administrators use to create applications and services. Notes databases store information and keep the extra function required to develop applications on the Lotus Domino platform.

IBM Lotus Domino Designer offers application development software that enables application developers and Web site designers to create security-rich, collaborative applications that can be accessed through Lotus Notes or supported Web browsers. Developers can easily create core business solutions based on several factors, including:

  • Forms
  • Views
  • Pages
  • Framesets
  • Integrated instant messaging
  • XML
  • Java™
  • JavaScript

Introducing Lotus Domino Web services

This section describes Lotus Domino and Web services.

Overview of Web services

A Web service is an archive of remote operations that can be called by sending messages over the Internet. A Web service provider publishes a Web service for query and use, and a Web service consumer calls operations from the service. A Web service provider makes available a Web Services Description Language (WSDL) document that defines the service interface. The WSDL document is in XML format. What happens behind the interface is up to the provider, but commonly providers map the interface to procedure calls in a supported programming language. Incoming requests from a consumer are passed through to the underlying code, and results are passed back to the consumer.

Lotus Domino and Web services

Lotus Domino V7.0 introduced the new Web service design element in Lotus Domino Designer. If you open a database in the Lotus Domino Designer V7.0 client, you see the Web Services entry just below the familiar Agents entry in the Shared Code section of the design element tree. Lotus Domino maps the WSDL interface to an agent-like Web service design element that can be coded in Lotus Script or Java. To be used, the Web service must be on a Domino server with HTTP enabled.

Lotus Domino handles all the WSDL creation and SOAP handling for you, so all you have to do is to write code in your Web service design element as though you were coding an agent. As long as you specify which class to use as the interface class for the service, Lotus Domino can publish a WSDL file, can convert incoming SOAP requests to method calls on your class, and can return the results of your method (if any) as a SOAP response. From a coding standpoint, all you do is write a Lotus Script or Java class. Lotus Domino does the rest

Creating a Lotus Domino Web Service

You can use any of several approaches to create a Web service design element in Domino Designer. If you code it entirely in Lotus Script or in Java, saving the design element generates a WSDL document that reflects the Lotus Script or Java code. If you import an existing WSDL document, Lotus Script or Java is generated that reflects the operations in the imported WSDL. The Web service design element saves the WSDL document and the code. If the public interface has not changed, the WSDL document stays as is. If, in your coding, you change anything that affects the public interface, a new WSDL is generated. In Domino Designer, the Web service design element resides below Agents under Shared Code.

  1. Ensure that in Domino Administrator, the Programmability Restrictions fields under Server document > Security tab have the names of the people who sign the Web services, including the administrator and the server. By default, these fields are blank, allowing no access.
  2. Launch Domino Designer.
  3. Create a new blank database by clicking File > Database > New or CTRL+N, or open an existing database by clicking File > Database > Open or CTRL+O. In the design view, you see that a new design element called Web Services is added to a Lotus Notes database, as shown in Figure 1.
Figure 1. Domino Designer and its Web service element
Screen cap: Design - Web Services window in                     Domino Designer showing Webservices tab selected
Screen cap: Design - Web Services window in Domino Designer showing Webservices tab selected
  1. Click New Web Service to generate a new Web service.

Coding the new Web service

  1. Code a class within this Web service design element using either Lotus Script or Java. All the methods of the class are automatically exposed as Web service methods.
  2. If you do not want a particular method to be exposed, change it to a Private Method by adding the Private qualifier before the function.

Basic coding

The code for a Web service has the following elements:

  • A class definition for the implementation code: this class must become the PortType class named in the Basics tab of the properties box, and the class must be public.
  • Within the class, a procedure (function, sub, or method) definition for each operation in the Web service: these procedures must be public. Supporting procedures that you don't want in the interface must be private.
  • Inclusion of Lotus Script .lss files or importing of java packages
  • Initialization of a Notes Session (LotusScript) or Session (Java) objects if Domino Objects are accessed.

Listing 1 and Figure 2 show how to create a very simple Lotus Script class that uses only one method to add two numbers and to return the result.

Listing 1. Sample Lotus Script code that adds two numbers
Class MathService
Function Sum(X as Integer, Y as Integer) As Integer
Sum = X+Y
End Function
End Class
Figure 2. Lotus Script to add two numbers
Screen cap: MathService - Web Service tab with                     Listing 1 content
Screen cap: MathService - Web Service tab with Listing 1 content

Setting Web Service properties

The full documentation to set properties is available in the Domino Designer documentation Related topics, so this article highlights only some of the important parts.

  1. Right-click the coding pane, and click Web Service Properties to bring up the properties dialog.
  2. Set the name of the Web Service Design element, such as Math.
  3. Specify which class is to be exposed as a Web service, such as MathService.
  4. Provide a name (required).
  5. Optionally provide an alias or comment.
  6. Optionally specify that you want to be warned if a coding change causes the generation of a new WSDL.
  7. Provide the PortType class, which is the name of the class that defines the procedures that map to the WSDL operations. These procedures must be public functions or subs in Lotus Script, or they must be public methods in Java. Private functions, subs, and methods are not exposed through the Web services interface. Note that you cannot enter the PortType class in the Properties window until you have created the class by coding or by importing a WSDL, as shown in Figure 3.
Figure 3. Web Service properties
Screen cap: MathService - Web Service tab with                     Web Service Properties window showing Name = MathService and                     PortType=MathService
Screen cap: MathService - Web Service tab with Web Service Properties window showing Name = MathService and PortType=MathService
  1. In the Security tab, specify that you want the Web service to run with the effective user name of the user invoking the Web service by checking Run as web user. For the simple Math example, you don't need to run any restricted operations, as shown in Figure 4.
Figure 4. Web Service properties
Screen cap:  Web Service properties window                     showing Run as Web User, Allow Remote Debugging, Profile This Web                     Service and All Readers and Above checked
Screen cap: Web Service properties window showing Run as Web User, Allow Remote Debugging, Profile This Web Service and All Readers and Above checked

The Advanced tab enables you to fine tune the Soap Message parameters to suit your needs, as shown in Figure 5.

Figure 5. SOAP message parameters on Advanced tab
Screen cap:  Web Service properties window                     Advanced tab showing Programming Model = RPC, SOAP Message Format =                     RPC/encoded, Include Op name in SOAP=checked, Port Type                     Name=MathService, Service Element Name=MathServiceService, and Service                     Port Name=Domino
Screen cap: Web Service properties window Advanced tab showing Programming Model = RPC, SOAP Message Format = RPC/encoded, Include Op name in SOAP=checked, Port Type Name=MathService, Service Element Name=MathServiceService, and Service Port Name=Domino

Accessing the Web service

The following are example URLs to access the Web service:

  • The WSDL definition for the Web service that you created is available at http://servername<:port>/databasename.nsf/webservicename?WSDL
  • In the example, this translates to http://9.182.77.93/webservi.nsf/MathService?WSDL
  • The Web service itself is available at http://servername<:port>/databasename.nsf/webservicename?OpenWebService
  • In the example, this translates to http://9.182.77.93/webservi.nsf/MathService?OpenWebService

Accessing the Lotus Domino Web service through Web Service Pack

High-level design architecture

This section describes how InfoSphere Information Server deployed on IBM WebSphere Application Server communicates with a Web service application deployed on Lotus Domino Server. The DataStage Web Services Pack constructs a SOAP request from the user-defined inputs and sends it over HTTP. The Domino application receives the SOAP request with the exposed Web service (the .nsf file that contains the Web service design element). The Domino Web service application processes the SOAP request and the result is passed to the DataStage Web Service Pack in the form of a SOAP response over HTTP. The Web Services Pack in turn parses the SOAP response, fetches the response value, and passes it to the DataStage user. For the example, the response value is logged to the DataStage Director. Figure 6 illustrates this architecture flow.

Figure 6. DataStage and Domino integration
Diagram showing (A) Requests and Responses going                     between Lotus Domino server tasks and the database with Web service                     design element. (B) DataStage Web service pack and Quality stage running                     any ETL tool. Then Requests and Responses going between A and B.
Diagram showing (A) Requests and Responses going between Lotus Domino server tasks and the database with Web service design element. (B) DataStage Web service pack and Quality stage running any ETL tool. Then Requests and Responses going between A and B.

Importing the WSDL metadata

You need to import message and metadata information from the WSDL document for each Web service operation that you want to call from a Web Services Pack stage. WSDL documents are parsed and converted to table definitions. To access and import WSDL documents, use the Web Service Metadata Importer.

  1. Open the InfoSphere DataStage Designer.
  2. Select Import > Table Definitions > Web Service Table Definitions, as shown in Figure 7. The Web Service Metadata Importer opens.
Figure 7. DataStage Designer
Screen cap: DataStage Designer showing selected                     menu options
Screen cap: DataStage Designer showing selected menu options
  1. Enter the URL for the WSDL document in the Address field. For the example, enter http://servername<:port>/databasename.nsf/webservicename?WSDL, as shown in Figure 8.
Figure 8. WSDL importer
Screen cap: Web Service Meta Data Importer window                     showing C:\Documents and                     Settings\Administrator\HYDCONDEVWIN01\Desktop\LotusPics\MathService.wsdl                     in the Address field
Screen cap: Web Service Meta Data Importer window showing C:\Documents and Settings\Administrator\HYDCONDEVWIN01\Desktop\LotusPics\MathService.wsdl in the Address field
  1. The Web Service Explorer window displays a tree view of several WSDL elements. Highlight one or more operations in the Web Service Explorer window.
  2. Right-click your selections, and select Import. The Import Progress window opens, as shown in Figure 9.
Figure 9. WSDL Importer: Import Progress
Screen cap: Import Progress window
Screen cap: Import Progress window
  1. Click Details to see the list of imported messages.
  2. Click Close to complete the import.

Setting up the Web Service Pack stage and the DataStage job

  1. From the Real Time category of the Palette window, drag the Web services client stage icon onto the canvas.
  2. Connect other stages based on whether the Web service acts as a data source or as a data target.
  3. Attach a target stage to the Web service stage to which the response of the Web service will be written. For the example, use a Peek Stage, which prints the Web service response in the director log, as shown in Figure 10.
Figure 10. DataStage parallel job
Drawing of Access_Lotus_DB linking using DSLink2                     to View_Result
Drawing of Access_Lotus_DB linking using DSLink2 to View_Result
  1. In the Web Services client stage window, click the General tab.
  2. Click Select Web Service Operation, as shown in Figure 11. The Web Service browser window opens.
Figure 11. Web service client stage
Screen cap: Access_Lotus_DB - WSClientPX stage                     window showing Select Web Service Operation button to select
Screen cap: Access_Lotus_DB - WSClientPX stage window showing Select Web Service Operation button to select
  1. In the Web services window, highlight the service name.
  2. In the Operations window, highlight the operation that you want to use in the Web services client stage.
  3. Click the Select this item link in the Information window. The Web services client stage window reappears. The Web service name and the selected operation name appear on the General window, as shown in Figure 12.
Figure 12. Web service operation name
Screen cap: Web Service Browser window with                     MathServiceService selected
Screen cap: Web Service Browser window with MathServiceService selected
Figure 13. Web Service properties
Screen cap: Access_Lotus_DB - WSClientPX stage                     window showing the Stage tab selected and Web Service information                     completed
Screen cap: Access_Lotus_DB - WSClientPX stage window showing the Stage tab selected and Web Service information completed
  1. Define the credentials needed to access the Web service using the Security Tab, as shown in Figure 14.
Figure 14. Single sign-on details
Screen cap: Access_Lotus_DB - WSClientPX stage                     window with Security tab selected and Authentication Required checked
Screen cap: Access_Lotus_DB - WSClientPX stage window with Security tab selected and Authentication Required checked
  1. On the Output Tab, click Input Arguments > Load Arguments, as shown in Figure 15. The input arguments to Web service are automatically loaded. A user enters the input values to those arguments.
Figure 15. Web service input arguments
Screen cap: Access_Lotus_DB - WSClientPX stage                     window with Output tab selected, Output Name=DSLink2 and Input                     Arguments tab is selected
Screen cap: Access_Lotus_DB - WSClientPX stage window with Output tab selected, Output Name=DSLink2 and Input Arguments tab is selected
  1. On the Output Message Tab, click Load Message Information, as shown in Figure 16. All the WSDL namespaces will be populated automatically.
Figure 16. Output message properties
Screen cap: Access_Lotus_DB - WSClientPX stage                     window with Output Message tab selected and Load Message Information                     button highlighted
Screen cap: Access_Lotus_DB - WSClientPX stage window with Output Message tab selected and Load Message Information button highlighted
  1. Go to the Columns tab to find the Web service response argument populated automatically, as shown in Figure 17.
Figure 17. Web service output arguments
Screen cap: Access_Lotus_DB - WSClientPX stage                     window with Output tab selected and Columns tabs selected
Screen cap: Access_Lotus_DB - WSClientPX stage window with Output tab selected and Columns tabs selected
  1. Attach a target stage to the Web service stage to which the response of the Web service will be written. For the example, attach Peek Stage, which prints the Web service response in the director log.

Compiling and running the DataStage job

  1. In the DataStage Designer, click the Compile button. A window pops up to show the successful compilation of the job, as shown in Figure 18.
Figure 18. Compile job
Screen cap: Compile Job - AccessDomino window                     showing the job compiled successfully
Screen cap: Compile Job - AccessDomino window showing the job compiled successfully
  1. Close the window, and click the Run button, as shown in Figure 19. The job runs successfully.
Figure 19. Run the job
Screen cap: AccessDomino - Job Run Options                     window with Run button selected
Screen cap: AccessDomino - Job Run Options window with Run button selected
  1. Open the DataStage Director, and check the job log. For the example, the Web service returns the sum of 300 in the director log, as shown in Figure 20.
Figure 20. Result in director log
Screen cap: Event Detail window showing                     View_Result=0; SumReturn=300 in the Message field
Screen cap: Event Detail window showing View_Result=0; SumReturn=300 in the Message field

Conclusion

The article started with an introduction to IBM InfoSphere Information Server, DataStage Web Services Pack, Lotus Domino, and Lotus Domino Web Services support. You learned how to create Web services using the Web Service Design Element in Domino Designer and how to deploy the Web services in Lotus Domino Server. Later you explored in detail how to configure a Web Service Client Stage in a DataStage Job to invoke the Lotus Domino Web Service.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Information Management, Collaboration, SOA and web services
ArticleID=455225
ArticleTitle=Use DataStage Web Services Pack to fetch Lotus Notes data into an InfoSphere Information Servers DataStage module for more ETL processing
publish-date=12102009