Skip to main content

By clicking Submit, you agree to the developerWorks terms of use.

The first time you sign into developerWorks, a profile is created for you. Select information in your developerWorks profile is displayed to the public, but you may edit the information at any time. Your first name, last name (unless you choose to hide them), and display name will accompany the content that you post.

All information submitted is secure.

  • Close [x]

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerworks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

By clicking Submit, you agree to the developerWorks terms of use.

All information submitted is secure.

  • Close [x]

Select a language:

  • Close [x]
  • Close [x]

High-performance solution to feeding a data warehouse with real-time data, Part 2: Explore the integration options with staging tables and WebSphere MQ messages

Integrating InfoSphere Replication Server and InfoSphere DataStage to trickle feed the data warehouse

Anand Krishniyer (akrishni@us.ibm.com), Staff Software Engineer, InfoSphere Replication, IBM
Anand Krishniyer photo
Anand Krishniyer is a staff engineer with the InfoSphere Replication development organization. As a member of the admin team, his responsibilities include developing line items as well as providing technical assistance to customers with respect to installation, setup, and configuration of the Replication Server. Prior to his current role, Anand worked as a project lead for the Integration and Tools team for a process management company, Savvion (now part of Progress Software).
Tony Lee (tonylee@us.ibm.com ), Senior Certified IT Specialist, IBM
Tony Lee photo
Tony Lee is a certified senior IT specialist in the InfoSphere Replication Center of Competency (COC), a team within the InfoSphere Replication development organization. As a member of the COC, Tony has provided technical assistance to customers and business partners on InfoSphere replication technologies. Tony has many years of consulting experience with various IBM replication technologies, covering SQL replication, Q replication, and, most recently, InfoSphere Change Data Capture. Prior to his current role, Tony provided Information Management technical consultation to customers and partners for nearly a decade, covering a wide range of topics, from DB2 tuning, to Information Server and Master Data Management. Prior to becoming a consultant, Tony worked in many different roles in the Information Management area, ranging from management to development.
James Yau (jamesyau@us.ibm.com), Technical Solution Architect, InfoSphere Information Server, IBM
James Yau photo
James Yau is a senior solution architect certified on the InfoSphere Information Server DataStage product. Currently, he is part of the InfoSphere Technology Enablement organization responsible for Information Server Boot Camp content development and delivery. James has many years of consulting experience with the Information Server Suite of products, including InfoSphere DataStage, QualityStage, Information Analyzer, and FastTrack. Prior to his current role, James was part of a Business Partner Technical Enablement team, in which he was the technical program manager for InfoSphere Information Server. His role included course content development and delivery with various delivery vehicles, such as instructor-led, on-line, and self-paced learning. In the past, James worked in many different roles, ranging from software developer to marketing manager, both in IBM and outside of IBM.

Summary:  Feeding a data warehouse with changes from the source database can be very expensive. If the extraction is only done with SQL, there is no way to easily identify the rows that have been changed. IBM InfoSphere™ Replication Server can detect changed data by reading only the database log. This series shows how to use InfoSphere Replication Server to efficiently extract only the changed data and how to pass the changes to IBM InfoSphere DataStage® to feed the data warehouse. Part 1 of the 2-part series provided an overview of these products and how they can work together. In this Part 2, explore two integration options: using WebSphere® MQ messages with InfoSphere Event Publisher and using staging tables.

View more content in this series

Date:  02 Sep 2010
Level:  Intermediate
PDF:  A4 and Letter (1877 KB | 52 pages)Get Adobe® Reader®

Activity:  17733 views
Comments:  

Before you start

About this tutorial

The Part 1 article of this series addressed the technologies of the InfoSphere Replication Server and DataStage products and the different ways of integrating the two to feed the warehouse. It also covered the pros and cons of the various integration options. In this Part 2 tutorial, explore two specific integration options: using MQ messages and using staging tables. This tutorial takes you through the setup and configuration of each of these integration options with screen shots and step-by-step instructions. This tutorial does not dive into the details of how to write a DataStage job or of how to configure replication, but instead concentrates on the integration techniques.

Back to top

Prerequisites

The scenarios described in the tutorial can be performed in the following environment:

Operating system and hardware
  • AIX®, Version 5.3, operating system with 64-bit Common Hardware Reference Platform (CHRP) architecture hardware
  • Windows XP Professional Service Pack 3 with 32-bit Intel processor
Software
  • IBM InfoSphere Replication Server 9.7
  • IBM Information Server 8.1 Server for AIX (with Connectors rollup patch 2 for DB2® connector)
  • IBM Information Server 8.1 Client for Windows® (with Connectors rollup patch 2 for DB2 connector)
  • WebSphere MQ, Version 7 (for the MQ scenario)

1 of 6 | Next

Comments

Back to top

Close [x]

Help: Update or add to My dW interests

What's this?

This little timesaver lets you update your My developerWorks profile with just one click! The general subject of this content (AIX and UNIX, Information Management, Lotus, Rational, Tivoli, WebSphere, Java, Linux, Open source, SOA and Web services, Web development, or XML) will be added to the interests section of your profile, if it's not there already. You only need to be logged in to My developerWorks.

And what's the point of adding your interests to your profile? That's how you find other users with the same interests as yours, and see what they're reading and contributing to the community. Your interests also help us recommend relevant developerWorks content to you.

View your My developerWorks profile

Return from help

Close [x]

Help: Remove from My dW interests

What's this?

Removing this interest does not alter your profile, but rather removes this piece of content from a list of all content for which you've indicated interest. In a future enhancement to My developerWorks, you'll be able to see a record of that content.

View your My developerWorks profile

Return from help

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Information Management, WebSphere
ArticleID=515022
TutorialTitle=High-performance solution to feeding a data warehouse with real-time data, Part 2: Explore the integration options with staging tables and WebSphere MQ messages
publish-date=09022010
author1-email=akrishni@us.ibm.com
author1-email-cc=
author2-email=tonylee@us.ibm.com
author2-email-cc=
author3-email=jamesyau@us.ibm.com
author3-email-cc=

Table of contents

1 of 6 | Next

Local resources

Tags

Help
Use the search field to find all types of content in My developerWorks with that tag.

Use the slider bar to see more or fewer tags.

Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).

My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).

Use the search field to find all types of content in My developerWorks with that tag. Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere). My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).

 

Dig deeper into Information Management on developerWorks

Try IBM PureSystems. No charge.

IBM PureSystems on a kaleideoscope background

Experience today!

Get started with the cloud-based trial and pattern development kit.

Share this page:

  • Close [x]

Follow developerWorks:

  • Close [x]