Introduction

IBM Directory Integrator had its technological beginnings addressing the synchronization problem with directories.

Directories such as IBM Directory Integrator Server or the Domino address book -- which has evolved into a general purpose directory -- contain information about people and resources that has been optimized for search and retrieval. To link the directories together, centralized meta directory solutions are often used. These solutions use centralized administrative functions that often lead to an unwieldy, hard-to-manage environment.

Directory Integrator supports LDAPv3, Java Naming and Directory Interface (JNDI), and product-specific connectors to access a wide variety of commercially available directories. It provides a powerful approach to the problem of corporate directory synchronization.

Data synchronization is one of the most perplexing and omnipresent problems with developing integrated e-business solutions. Within large, distributed, and often complex systems, customer data is in several locations and might be maintained by different software packages. For example, within an e-commerce solution, customer data might be within the e-commerce system itself, an associated customer relationship management (CRM ) system, and a supply chaining management (SCM) system. Many approaches to solving this problem have been proposed over the years, but they require complex, difficult-to-maintain programming.

The IBM Business Integration initiative (see Resources for a link) addresses these issues with several products, such as WebSphere Business Integrator, which can bring about integration at a business process level. You don't have to worry about the particulars of the underlying data store, which might not even reside within your environment. Development is done at the business process level, with a business process interacting with one or more other business processes.

In this article we explore how Directory Integrator provides basic data integration for a project. We show Directory Integrator's very simple way to encapsulate a wide range of protocols, formats, APIs, and scripting languages by using connectors.

Note: Directory Integrator will run on many platforms, including Windows, Linux, Solaris, HP-UX, AIX, and Macintosh X. Directory Integrator provides a Java virtual machine that conforms to the Java 2 specification. We ran the examples in this article on Windows 2000.

Back to top

IBM Directory Integrator

The Directory Integrator development paradigm manifests itself in AssemblyLines. An AssemblyLine, shown in Figure 1 below, describes the origin, transformation, and destination of data. Integration elements, including data sources, events, data flows, and attribute mapping with transformation, are used to create the assemblies, or chains.

In the figure, the data enters the AssemblyLine from the left and ends its journey at a data store in the upper right. Based on the configuration of individual event handlers, connectors, event parsers, data parsers, hooks and business rules, an AssemblyLine executes the prescribed data flows.

An event handler detects a predefined event and applies an [event+condition--> action] paradigm. At that point, an event parser may provide interpretation and transformation functions to the data flow. Connectors along the AssemblyLine execution flow provide access to a number of devices, systems, applications and platforms. Directory Integrator delivers many connectors, each designed for well known data sources such as DB2 and LDAP.

Also available for deployment along the AssemblyLine are parsers and hooks. Parsers facilitate the interpretation and translation of information byte stream data into structured "information objects." Hooks provide the ability to execute actions under specific conditions or at specific points along the AssemblyLine. With additional Java code or by using scripting languages such as JavaScript, Perl and VBScript additional business rules and real-time event handling can be added.

As shown in the following sections, the Directory Integrator architecture allows for development by accretion. Intermediate steps can be immediately tested and debugged. Additional logic can be added quickly to the assembly to create the next revision.

Back to top

Our integrated solution environment

Each summer IBM runs Extreme Blue, a high-energy college intern program (see Resources for more information). Last summer our Extreme Blue project, codenamed "bluebiz," built a prototype service-level management (SLM) system for service providers. Figure 2 shows a sample SLM architecture.

Back to top

Transforming and synchronizing with Directory Integrator

An SLM solution is comprised of many functions, including metering, monitoring, and billing. Data related to these activities would usually reside in disparate data stores such as databases, comma separated values (CSV) files, XML, and so on. You can use Directory Integrator to provide a clean, simple solution to address data synchronization issues within this environment.

Let's look at a simple example from our project that shows how Directory Integrator can be used. Within our SLM solution there are service providers and service consumers. The service providers provide "service offerings" that are agreements to provide services at a given QOS. A service consumer subscribes to service offerings, which might be as simple as a credit card validation service. Once the agreement is in place, the SLM system monitors the actual service delivered and compares it with the agreed upon terms. If there is a violation, a suitable action is taken to remedy the problem, and if needed the requisite remediation occurs at billing time.

We'll assume that service offering, metering, and billing related data is stored in DB2 tables: sladb (slaid,slaname,guarantee), metdb(slaid,custid,value), and billdb(custid,slaid,charge) respectively. Customer-related data is in a CSV file (custid,custname,slaid). The consolidated output is expected to be an XML file.

Note: To run this example, you must have both IBM DB2 Universal Database and Directory Integrator installed.

Step 1: Setting up the environment

• Install the version of Directory Integrator for your platform, and configure per the instructions.
• Install DB2 and run the db2scripts.bat file provided. It will create the sample database.
• Bring up the Directory Integrator administration window.

Step 2: Creating a new configuration and assembly line

• By default an rs.cfg is opened. Click File>New and create a new configuration file named metamergetut.cfg.
• Create a new AssemblyLine by clicking Add as shown in Figure 3 below.
  Figure 3. Adding an AssemblyLine
  
  

Step 3: Creating CSV input connector

• Click Add to add a new input connector for the customer data, which is a CSV file. The add connector dialog box, in Figure 4 below, has three input fields:
  • Name: Enter CSVInput.
  • Connector: Select MetaMerge.FileSystem.
  • Connector Mode: Select Iterator, because we are going to iterate over each entry in the CSV file.
  
  Figure 4. Adding a connector
  
  
• Click OK, then Configure. A new dialog box pops up that lets you configure the settings of the connector we just added.
  Figure 5. Configuring a connector
  
  
• As shown in Figure 5, select the File path to point to inputcsv.csv.
• Select Input as the mode.
• Select Metamerge.CSV in the Content Parser because we are dealing with a CSV file. Next click Attributes.
  Figure 6. Setting the connector attributes
  
  
• Click on Connect then Next, and a table showing the contents of the CSV file is displayed, as in Figure 6 above. The first column shows the name of the attribute, the second the data type that Directory Integrator uses to interpret the data, and third is the value. Click Close and return to the main screen. Now we have set up the input connector to iterate over the contents of the CSV file.
• Click on Attribute Map, which displays the list of attributes that were obtained by reading the CSV file. Select all the attributes. These now become the working attributes, which can be used internally by the Directory Integrator assembly, as you'll soon see.

Step 4: Configure connectors connecting the billing, metering and service offering tables so we can lookup on them using the data obtained by iterating over the CSV file.

• Click Add to add a new connector for the service offering table (sladb). Enter a name for the connector, then select MetaMerge.JDBC as the connector and select Lookup as the mode. Click OK.
• Click Configure and enter the details, as shown in Figure 7 below. Be sure you enter the appropriate user name and passwords.
  Figure 7. Configuring the database connector
  
  
• Click Attributes, then Connect, and then Next. A table similar to the previous one shows the contents of the table. Click Close.
• Click Link Criterion, which lets you do lookups on the table based on some working attributes you already may have -- in our case, the custid and slaid attributes we obtained from the CSV file.
  Figure 8. Setting up the Lookup
  
  
• As shown above in Figure 8, select SLAID as the Connector Attribute, equals as the Operand, and $slaid as the Value. Click OK.
• Click on the Attribute Map tab, select SLAName and Guarantee, then click OK. Now you have set up a connector to lookup into the sladb based on the values of custid and slaid obtained from the CSV connector. Repeat these steps for the metering table and billing table.

Step 5: Configuring the XML output connector

• Now that you've gotten data from all your sources, you need to consolidate and output it in an XML file. Click Add to add a new connector. As shown in Figure 9, set the Name, select MetaMerge.filesystem as the Connector, and Addonly as the Mode.
  Figure 9. Creating the output connector
  
  
• Click Configure. Set the file path to the directory/filename.xml where the file is to be generated. Select Output as the mode and MetaMerge.XML as the content parser. Click OK.
• Click Attribute Map and choose Select. Select custname, value, slaname, charge and guarantee. Click OK.

Step 6: Run the Assembly

• Now you're set to run your assembly. The CSV input connector will iterate over the contents of the CSV file. For each iteration the Sladblookup, billdblookup and metdblookup connectors will carry out a lookup based on the slaid and/or custid obtained from the input connector. The results of the lookup will then be put in the output XML connector, which will then write it to the file. Click on Run. The assembly will execute and the XML file will be generated, as shown in Figure 10.

Back to top

Summary

This article described the basics of using an AssemblyLine and connectors to map and transform data from a CSV file into a relational database system. We've only scratched the surface of what is possible. IBM Directory Integrator provides a powerful and flexible environment for data transformation and synchronization.

Back to top

Download

Information about download methods

Resources

• Download a zip file that contains db2script.bat, used to create the sample database used in this article.
  
  
• Learn more about IBM Directory Integrator.
  
  
• Get information on DB2 or on WebSphere Business Integration Software.
  
  
• Download a trial copy of DB2 UDB Personal Developer's Edition Version 7.2.
  
  
• Read "From Data Management to Information Integration: A Natural Evolution" (DB2 Developer Domain, June 2002).
  
  
• Get more information on IBM Business Integration.
  
  
• Read about Extreme Blue.
  
  

About the authors

Rate this article

Comments

Back to top

Table of contents

• Introduction
• IBM Directory Integrator
• Our integrated solution environment
• Transforming and synchronizing with Directory Integrator
• Summary
• Download
• Resources
• About the authors
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America