The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerworks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

All information submitted is secure.

My developerWorks:

My notifications:

Sign out

Select a language:

High-performance solution to feeding a data warehouse with real-time data, Part 2: Explore the integration options with staging tables and WebSphere MQ messages

Integrating InfoSphere Replication Server and InfoSphere DataStage to trickle feed the data warehouse

Anand Krishniyer (akrishni@us.ibm.com), Staff Software Engineer, InfoSphere Replication, IBM

Anand Krishniyer is a staff engineer with the InfoSphere Replication development organization. As a member of the admin team, his responsibilities include developing line items as well as providing technical assistance to customers with respect to installation, setup, and configuration of the Replication Server. Prior to his current role, Anand worked as a project lead for the Integration and Tools team for a process management company, Savvion (now part of Progress Software).

Tony Lee (tonylee@us.ibm.com ), Senior Certified IT Specialist, IBM

Tony Lee is a certified senior IT specialist in the InfoSphere Replication Center of Competency (COC), a team within the InfoSphere Replication development organization. As a member of the COC, Tony has provided technical assistance to customers and business partners on InfoSphere replication technologies. Tony has many years of consulting experience with various IBM replication technologies, covering SQL replication, Q replication, and, most recently, InfoSphere Change Data Capture. Prior to his current role, Tony provided Information Management technical consultation to customers and partners for nearly a decade, covering a wide range of topics, from DB2 tuning, to Information Server and Master Data Management. Prior to becoming a consultant, Tony worked in many different roles in the Information Management area, ranging from management to development.

James Yau (jamesyau@us.ibm.com), Technical Solution Architect, InfoSphere Information Server, IBM

James Yau is a senior solution architect certified on the InfoSphere Information Server DataStage product. Currently, he is part of the InfoSphere Technology Enablement organization responsible for Information Server Boot Camp content development and delivery. James has many years of consulting experience with the Information Server Suite of products, including InfoSphere DataStage, QualityStage, Information Analyzer, and FastTrack. Prior to his current role, James was part of a Business Partner Technical Enablement team, in which he was the technical program manager for InfoSphere Information Server. His role included course content development and delivery with various delivery vehicles, such as instructor-led, on-line, and self-paced learning. In the past, James worked in many different roles, ranging from software developer to marketing manager, both in IBM and outside of IBM.

Summary: Feeding a data warehouse with changes from the source database can be very expensive. If the extraction is only done with SQL, there is no way to easily identify the rows that have been changed. IBM InfoSphere™ Replication Server can detect changed data by reading only the database log. This series shows how to use InfoSphere Replication Server to efficiently extract only the changed data and how to pass the changes to IBM InfoSphere DataStage® to feed the data warehouse. Part 1 of the 2-part series provided an overview of these products and how they can work together. In this Part 2, explore two integration options: using WebSphere® MQ messages with InfoSphere Event Publisher and using staging tables.

View more content in this series

Date: 02 Sep 2010
Level: Intermediate
PDF: A4 and Letter (1877 KB | 52 pages)Get Adobe® Reader®

Activity: 23370 views
Comments:

Before you start

About this tutorial

The Part 1 article of this series addressed the technologies of the InfoSphere Replication Server and DataStage products and the different ways of integrating the two to feed the warehouse. It also covered the pros and cons of the various integration options. In this Part 2 tutorial, explore two specific integration options: using MQ messages and using staging tables. This tutorial takes you through the setup and configuration of each of these integration options with screen shots and step-by-step instructions. This tutorial does not dive into the details of how to write a DataStage job or of how to configure replication, but instead concentrates on the integration techniques.

Prerequisites

The scenarios described in the tutorial can be performed in the following environment:

Operating system and hardware

AIX®, Version 5.3, operating system with 64-bit Common Hardware Reference Platform (CHRP) architecture hardware
Windows XP Professional Service Pack 3 with 32-bit Intel processor

Software

IBM InfoSphere Replication Server 9.7
IBM Information Server 8.1 Server for AIX (with Connectors rollup patch 2 for DB2® connector)
IBM Information Server 8.1 Client for Windows® (with Connectors rollup patch 2 for DB2 connector)
WebSphere MQ, Version 7 (for the MQ scenario)

Comments

static.content.url=http://www.ibm.com/developerworks/js/artrating/

SITE_ID=1

Zone=Information Management, WebSphere

ArticleID=515022

TutorialTitle=High-performance solution to feeding a data warehouse with real-time data, Part 2: Explore the integration options with staging tables and WebSphere MQ messages

publish-date=09022010

author1-email=akrishni@us.ibm.com

author1-email-cc=

author2-email=tonylee@us.ibm.com

author2-email-cc=

author3-email=jamesyau@us.ibm.com

author3-email-cc=