Use DataStage Web Services Pack to fetch Lotus Notes data into an InfoSphere Information Servers DataStage module for more ETL processing
A step-by-step guide
InfoSphere® Information Server's DataStage® module provides many operators and plug-ins for various data sources such as Oracle, DB2®, XML, Teradata, and SAP. But the DataStage module does not include a dedicated plug-in or operator to fetch Lotus Notes® data into DataStage to perform ETL operations.
Many customers have asked for this function. The common requirement is to fetch Notes data into DataStage, perform an extract, transform, and load (ETL), and then pass this data to other Information Server modules for more ETL processing. You cannot use the Lotus SQL driver, because it has issues on UNIX®-based systems due to incompatibilities with the latest ODBC standards. You cannot use third-party ODBC-to-ODBC bridges, because they are licensed. A cost-effective and novel way to fetch Lotus Notes data into InfoSphere Information Servers DataStage module for more ETL processing is needed.
This article explores various features that Lotus Domino® Server provides to create and deploy Lotus applications as Web services. You will learn how IBM InfoSphere Information Server uses these applications through DataStage Web Services Pack. This step-by-step guide helps you create, configure, and deploy Web services in Lotus Domino, and it helps you create, configure, compile, and execute a DataStage Web Services Pack job that uses Lotus Web services.
Introducing the products
This section describes the IBM Information Server, DataStage Web Services Pack, Lotus Domino, and the Lotus Domino suite of products.
IBM Information Server and the DataStage Web Services Pack
IBM InfoSphere Information Server is a unified and comprehensive information integration platform. It profiles, cleanses, and transforms data from heterogeneous data sources to deliver consistent and accurate business data. IBM Information Server is an ideal solution to integrate and synchronize XML data with the business data stored within back-end enterprise resource planning (ERP) systems and other data analytical applications.
Using the DataStage Web Services Pack, you can access Web service operations within an IBM InfoSphere Information Server DataStage job. The Web Services Pack includes plug-in stages and added function for the DataStage server routines.
Lotus Domino and its suite of products
As integral parts of the IBM Workplace family, IBM Lotus Notes and Domino software enable users to share, manage, and organize information more efficiently. Lotus Notes and Domino 7 software products deliver a reliable, security-rich messaging and collaborative environment that can help companies enhance people's productivity, streamline business processes, and improve overall business responsiveness. Lotus Domino is an application server that provides groupware functions and development tools to create messaging, collaboration, workflow, tracking, Internet, and intranet applications. The Domino server stores information in Lotus Notes databases. All information, including POP3, HTTP, and Web services, resides in these Notes databases, which are the building blocks developers and administrators use to create applications and services. Notes databases store information and keep the extra function required to develop applications on the Lotus Domino platform.
IBM Lotus Domino Designer offers application development software that enables application developers and Web site designers to create security-rich, collaborative applications that can be accessed through Lotus Notes or supported Web browsers. Developers can easily create core business solutions based on several factors, including:
- Integrated instant messaging
Introducing Lotus Domino Web services
This section describes Lotus Domino and Web services.
Overview of Web services
A Web service is an archive of remote operations that can be called by sending messages over the Internet. A Web service provider publishes a Web service for query and use, and a Web service consumer calls operations from the service. A Web service provider makes available a Web Services Description Language (WSDL) document that defines the service interface. The WSDL document is in XML format. What happens behind the interface is up to the provider, but commonly providers map the interface to procedure calls in a supported programming language. Incoming requests from a consumer are passed through to the underlying code, and results are passed back to the consumer.
Lotus Domino and Web services
Lotus Domino V7.0 introduced the new Web service design element in Lotus
Domino Designer. If you open a database in the Lotus Domino Designer V7.0
client, you see the
Web Services entry just
below the familiar
Agents entry in the
Shared Code section of the design element tree.
Lotus Domino maps the WSDL interface to an agent-like Web service design
element that can be coded in Lotus Script or Java. To be used, the Web
service must be on a Domino server with HTTP enabled.
Lotus Domino handles all the WSDL creation and SOAP handling for you, so all you have to do is to write code in your Web service design element as though you were coding an agent. As long as you specify which class to use as the interface class for the service, Lotus Domino can publish a WSDL file, can convert incoming SOAP requests to method calls on your class, and can return the results of your method (if any) as a SOAP response. From a coding standpoint, all you do is write a Lotus Script or Java class. Lotus Domino does the rest
Creating a Lotus Domino Web Service
You can use any of several approaches to create a Web service design
element in Domino Designer. If you code it entirely in Lotus Script or in
Java, saving the design element generates a WSDL document that reflects
the Lotus Script or Java code. If you import an existing WSDL document,
Lotus Script or Java is generated that reflects the operations in the
imported WSDL. The Web service design element saves the WSDL document and
the code. If the public interface has not changed, the WSDL document stays
as is. If, in your coding, you change anything that affects the public
interface, a new WSDL is generated. In Domino Designer, the Web service
design element resides below
- Ensure that in Domino Administrator, the Programmability Restrictions fields under Server document > Security tab have the names of the people who sign the Web services, including the administrator and the server. By default, these fields are blank, allowing no access.
- Launch Domino Designer.
- Create a new blank database by clicking File > Database > New or CTRL+N, or open an existing database by clicking File > Database > Open or CTRL+O. In the design view, you see that a new design element called Web Services is added to a Lotus Notes database, as shown in Figure 1.
Figure 1. Domino Designer and its Web service element
- Click New Web Service to generate a new Web service.
Coding the new Web service
- Code a class within this Web service design element using either Lotus Script or Java. All the methods of the class are automatically exposed as Web service methods.
- If you do not want a particular method to be exposed, change it to a Private Method by adding the Private qualifier before the function.
The code for a Web service has the following elements:
- A class definition for the implementation code: this class must become
the PortType class named in the
Basicstab of the properties box, and the class must be public.
- Within the class, a procedure (function, sub, or method) definition for each operation in the Web service: these procedures must be public. Supporting procedures that you don't want in the interface must be private.
- Inclusion of Lotus Script .lss files or importing of java packages
- Initialization of a Notes Session (LotusScript) or Session (Java) objects if Domino Objects are accessed.
Listing 1 and Figure 2 show how to create a very simple Lotus Script class that uses only one method to add two numbers and to return the result.
Listing 1. Sample Lotus Script code that adds two numbers
Class MathService Function Sum(X as Integer, Y as Integer) As Integer Sum = X+Y End Function End Class
Figure 2. Lotus Script to add two numbers
Setting Web Service properties
The full documentation to set properties is available in the Domino Designer documentation Related topics, so this article highlights only some of the important parts.
- Right-click the coding pane, and click Web Service Properties to bring up the properties dialog.
- Set the name of the Web Service Design element, such as
- Specify which class is to be exposed as a Web service, such as
- Provide a name (required).
- Optionally provide an alias or comment.
- Optionally specify that you want to be warned if a coding change causes the generation of a new WSDL.
- Provide the PortType class, which is the name of the class that defines the procedures that map to the WSDL operations. These procedures must be public functions or subs in Lotus Script, or they must be public methods in Java. Private functions, subs, and methods are not exposed through the Web services interface. Note that you cannot enter the PortType class in the Properties window until you have created the class by coding or by importing a WSDL, as shown in Figure 3.
Figure 3. Web Service properties
- In the Security tab, specify that you want the Web service to run with the effective user name of the user invoking the Web service by checking Run as web user. For the simple Math example, you don't need to run any restricted operations, as shown in Figure 4.
Figure 4. Web Service properties
The Advanced tab enables you to fine tune the Soap Message parameters to suit your needs, as shown in Figure 5.
Figure 5. SOAP message parameters on Advanced tab
Accessing the Web service
The following are example URLs to access the Web service:
- The WSDL definition for the Web service that you created is available
- In the example, this translates to
- The Web service itself is available at
- In the example, this translates to
Accessing the Lotus Domino Web service through Web Service Pack
High-level design architecture
This section describes how InfoSphere Information Server deployed on IBM WebSphere Application Server communicates with a Web service application deployed on Lotus Domino Server. The DataStage Web Services Pack constructs a SOAP request from the user-defined inputs and sends it over HTTP. The Domino application receives the SOAP request with the exposed Web service (the .nsf file that contains the Web service design element). The Domino Web service application processes the SOAP request and the result is passed to the DataStage Web Service Pack in the form of a SOAP response over HTTP. The Web Services Pack in turn parses the SOAP response, fetches the response value, and passes it to the DataStage user. For the example, the response value is logged to the DataStage Director. Figure 6 illustrates this architecture flow.
Figure 6. DataStage and Domino integration
Importing the WSDL metadata
You need to import message and metadata information from the WSDL document for each Web service operation that you want to call from a Web Services Pack stage. WSDL documents are parsed and converted to table definitions. To access and import WSDL documents, use the Web Service Metadata Importer.
- Open the InfoSphere DataStage Designer.
- Select Import > Table Definitions > Web Service Table Definitions, as shown in Figure 7. The Web Service Metadata Importer opens.
Figure 7. DataStage Designer
- Enter the URL for the WSDL document in the Address field. For the
http://servername<:port>/databasename.nsf/webservicename?WSDL, as shown in Figure 8.
Figure 8. WSDL importer
- The Web Service Explorer window displays a tree view of several WSDL elements. Highlight one or more operations in the Web Service Explorer window.
- Right-click your selections, and select Import. The Import Progress window opens, as shown in Figure 9.
Figure 9. WSDL Importer: Import Progress
- Click Details to see the list of imported messages.
- Click Close to complete the import.
Setting up the Web Service Pack stage and the DataStage job
- From the Real Time category of the Palette window, drag the Web services client stage icon onto the canvas.
- Connect other stages based on whether the Web service acts as a data source or as a data target.
- Attach a target stage to the Web service stage to which the response of the Web service will be written. For the example, use a Peek Stage, which prints the Web service response in the director log, as shown in Figure 10.
Figure 10. DataStage parallel job
- In the Web Services client stage window, click the General tab.
- Click Select Web Service Operation, as shown in Figure 11. The Web Service browser window opens.
Figure 11. Web service client stage
- In the Web services window, highlight the service name.
- In the Operations window, highlight the operation that you want to use in the Web services client stage.
- Click the Select this item link in the Information window. The Web services client stage window reappears. The Web service name and the selected operation name appear on the General window, as shown in Figure 12.
Figure 12. Web service operation name
Figure 13. Web Service properties
- Define the credentials needed to access the Web service using the Security Tab, as shown in Figure 14.
Figure 14. Single sign-on details
- On the Output Tab, click Input Arguments > Load Arguments, as shown in Figure 15. The input arguments to Web service are automatically loaded. A user enters the input values to those arguments.
Figure 15. Web service input arguments
- On the Output Message Tab, click Load Message Information, as shown in Figure 16. All the WSDL namespaces will be populated automatically.
Figure 16. Output message properties
- Go to the Columns tab to find the Web service response argument populated automatically, as shown in Figure 17.
Figure 17. Web service output arguments
- Attach a target stage to the Web service stage to which the response of the Web service will be written. For the example, attach Peek Stage, which prints the Web service response in the director log.
Compiling and running the DataStage job
- In the DataStage Designer, click the Compile button. A window pops up to show the successful compilation of the job, as shown in Figure 18.
Figure 18. Compile job
- Close the window, and click the Run button, as shown in Figure 19. The job runs successfully.
Figure 19. Run the job
- Open the DataStage Director, and check the job log. For the example, the Web service returns the sum of 300 in the director log, as shown in Figure 20.
Figure 20. Result in director log
The article started with an introduction to IBM InfoSphere Information Server, DataStage Web Services Pack, Lotus Domino, and Lotus Domino Web Services support. You learned how to create Web services using the Web Service Design Element in Domino Designer and how to deploy the Web services in Lotus Domino Server. Later you explored in detail how to configure a Web Service Client Stage in a DataStage Job to invoke the Lotus Domino Web Service.
- Check out "Practical Web services in IBM Lotus Domino 7: What are Web services and why are they important?" (developerWorks 2006 Nov)
- Read "Practical Web services in IBM Lotus Domino 7: Writing complex Web services," (developerWorks 2006 Nov)
- Review "Lotus Notes/Domino 7 Web services" (developerWorks 2004 Sep)
- Get the resources you need to advance your skills in the architecture arena at the InfoSphere area on developerWorks.
- Get more details from the IBM Information Server information center.
- Download IBM product evaluation versions of Lotus Domino and its suite of products