This post is the answer to one of the FAQs found in License Tips for IBM Data Replication.
You may have seen a recent announcement on ibm.com
that says IBM would no longer be marketing it's older data replication products in 2013. That includes InfoSphere CDC. Why?
And what happens to the CDC technology?
Over the years, IBM provided its data replication technologies
through a lot of different products. For example, IBM used to offer two major
data replication products at the same time -
InfoSphere CDC and InfoSphere Replication Server. That was a little
even to some IBM people. To simplify the situation, IBM consolidated all it's replication technologies into a single product called IBM InfoSphere Data Replication
(IIDR). Once IIDR was available, the older products no longer needed to be sold to new customers. That's why the end of marketing was announced. However, the replication technologies - CDC, Q Replication, and SQL Replication - are still alive and well. You can
continue to use them as you always
have. Of course, you may have two related questions:
- Are the older products still being supported?
- How do you move from your old InfoSphere CDC product to IIDR?
If you have any questions, feel free to post them in the comments section of this blog.
I have updated the look and feel of the wiki to make it easier to navigate and find what you need.
Samples are now organized in their own table and can be found by clicking the "Samples" icon on the home page.
All "how-to" documents can now be found by clicking on the "Documents" icon on the home page.
A new section has been added to the wiki which describes the features that have been added to all the recent IIDR CDC releases. In the near future I will be adding presentations with details on the features in this section.... so please check back.
The IBM Developer Works Data Replication communities are being merged into this community. As such, the "The CDC (Change Data Capture) Forum" is being renamed to better represent the expanded content that will now reside in this community.
"The CDC (Change Data Capture) Forum" is being renamed to "IBM Data Replication Community".
This community will include content from what was formerly:
- The CDC (Change Data Capture) Forum
- The Q+SQL Replication Forum
I'm recording some videos where I provide technical background about data replication. I've created a YouTube channel to collect them all together. The channel is "James Talks about Data Replication". Here's a link:
I've uploaded two so far. The first one discusses the special considerations you should be aware of when using data replication with tables that have duplicate rows. The second discusses the role of data replication when moving to a real time operational analytics system from a traditional batch oriented data warehouse.
In response to: Best Practice - Target Considerations
Regarding Oracle triggers on the target tables, would these fire
during a standard refresh? We want them to fire off during normal
continuous replication from DB2 to Oracle..
Most are triggers on insert so I am concerned during the refresh
which we do whenever we change the table structure to match the
The new IIDR 11.3.3 Release is available today!
Here is a link to a page that describes the new functionality available: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W8d78486eafb9_4a06_a482_7e7962f5ac59/page/IIDR%20CDC%20New%20Features%20By%20Release?section=IIDR%2011.3.3
There are a large number of usability enhancements, new platform/DB support such as SQL Server 2014, along with technology previews/beta available for many new key technologies such as a Cloudant apply, WebHDFS support for Hadoop, and many others.
For those who have been using the CDC Developer Works Community, you will notice a change to the look & feel of the entry page. This new menu driven mechanism will provide extra flexibility as significant new content gets added to this community. I have tested it out on various browsers, so if you experience an issue, please send me an email at email@example.com and I will investigate.
Note I will also be cleaning up some old blog entries where questions were asked. Questions should continue to be asked on the forum versus the blog.
Hope you continue to get benefit from this community.
Modified on by DavidT
In April 2013, IBM announced Version 10.5 of DB2 for Linux, UNIX, and Windows. The same letter announced that DB2 AESE and DB2 AWSE would provide limited use of IBM InfoSphere Data Replication's (IIDR's) Change Data Capture (CDC) technology at no additional cost. However, the "limited use" statement sometimes leaves people with a question or two. The goal of this post is to answer those questions.
First, what CDC function are you entitled to use in the DB2 Advanced Editions? The license is the always final word, but, in simple terms, you can only use the bundled CDC to build disaster recovery solutions where a primary DB2 instance* has up-to-two backup instances. For example, the following replication topology is allowed by the DB2 Advanced Edition licenses:
Furthermore, the disaster recovery use case limits your entitled use of CDC function in the following ways:
You can only use unidirectional (one-way) replication.
You can set up replication from the primary DB2 to the backup(s) but you cannot set up replication from the backup(s) to the primary. This fits with the definition of a pure disaster recovery solution since it provides for fail-over but not switchback. If you need CDC for both fail-over and switchback, you need to license the full IIDR product.
You cannot transform the data as it's replicated. Again, this fits with the definition of disaster recovery and you can license the full IIDR to be entitled to transformations as you replicate.
The question is - when do you need to buy CDC now? If you want to do anything more than what's described in this post, you'll need to buy IIDR for your DB2 Advanced Editions. The two most common replication configurations that require this are ones where you do either of the following:
Replicate between DB2 LUW and either DB2 z/OS or Oracle.
Set up an HA or Active-Active solution with IIDR's CDC technology.
If you need to understand more about these examples, we'll have pictures and add a few more examples in a future post that talks about when you need to buy CDC.
Of course, the last question is - can I still build DB2 DR, HA, and Active-Active solutions using the Q Replication built into the DB2 Advanced Editions? Yes, absolutely. The addition of CDC to DB2 does not change this.
* Multiple DB2 instances can be created from a single DB2 install. Each instance can use the bundled CDC to replicate up to the entitled number of backup instances.
Modified on by GlenSakuth
I have had many requests to share best practices when using IBM InfoSphere Change Data Capture (from this point forward in the blog referred to as CDC). I will try to add new tips and techniques on a regular basis.
Along with many of the best practices posts, I will include items denoted by "Rule of Thumb". These are general guidelines that will help in your planning. I will endeavor to provide reasons or context for the guidance. The Rules of Thumb should not be treated as hard limits, rather as useful guidance. If your needs fall significantly outside the guidance, it certainly does not mean that it can not be done. Rather, it would be best to engage with an InfoSphere CDC subject matter expert, and potentially you may want to consider IBM Services for assistance.
Steady State Operations
Modified on by MikeJory
IIDR 220.127.116.11 has been released with new functionality in addition to resolving some APARs.
IIDR 18.104.22.168-4 can be found on Fix Central.
* No IIDR downtime support for Microsoft SQL Server non-structural online index rebuild operations
Index and table reorganizations done to improve query performance no longer impact replication. Replication will continue with no outage and with no user actions required.
* Log based change capture for Oracle is now supported on Microsoft Windows
* No IIDR downtime support for Oracle Automatic Storage Management rebalancing activities
Previously in some situations rebalancing of ASM could could require replication be restarted
* An unplanned failure of Oracle to a DataGuard standby database no longer requires a resynchronization
Previously if an Oracle database being used as a source of changes failed over to an Oracle DataGuard standby, you would have been required to refresh the target data before restarting replication
* IIDR can now capture changes directly from ASM on Exadata
Previously replication required the ASM logs to be multiplexed to a non-ASM destination which typically could only be accomodated off the Exadata appliance
* IIDR can now target Cloudant
* IIDR can now use webHDFS to deliver change data to HDFS
This allows IIDR to be installed remotely from the cluster and enables targeting HDFS deployments that are secured via Kerberos
JR53019 POTENTIAL FOR DUPLICATE TRANSACTION REPLAY ON THE TARGET DUE TO INCORRECT BOOKMARKING ON IMMEDIATELY PREVIOUS SHUTDOWN.
JR53294 CDC TRIES TO READ AN UNCOMPLETED ORACLE LOG AND GETS ERROR IN LOG SHIPPING USING DATAGUARD SCENARIO.
JR53370 CDC DATA LOSS / CONSISTENCY UPOEN CDC RESTART AFTER ABORT
JR52825 CDC FOR DATASTAGE IN DIRECTCONNECT MODE NOT HONOURING THE DELAY PARAMETER FROM BATCH THRESHOLD SUBS PROPERTY
JR52973 DMSHUTDOWN COMMAND WITH -A OPTION DOESN'T TERMINATE SUBSCRIPTIONS
JR52797 ADDRESSING AN IDENTIFIED BUFFER OVERRUN CONDITION IN THE ORACLE SOLARIS JVM
JR53411 SETTING BLOB/CLOB TRUNCATION IN DATASTAGE PROPERTIES FOR A SUBSCRIPTION, REVERSES THE LIMITS
JR52923 TIMESTAMP IS NOT PROPERLY REPLICATING
JR53413 HAVING ROW SIZES GREATER THAN 341333 BYTES THROWS JAVA.NIO.BUFFEROVERFLOWEXCEPTION
JR52946 INSTANCE USING UPPER CASE ORACLE SID FOR STANDBY DATABASE CANNOT BE CREATED
JR53340 IIDR FOR ORACLE 12C FAILS TO START REPLICATION WHEN AN INSCOPE TABLE WAS ALTERED
Now that IBM has packaged its major data replication technologies into a single product, InfoSphere Data Replication
, a lot of people are asking what they can take advantage of that they couldn't with the older products (InfoSphere CDC and InfoSphere Replication Server). Other than the obvious point of having access to multiple technologies, you can now use IBM's table compare utility, asntdiff
, with CDC. asntdiff is a general-purpose utility that compares the data from two queries. IBM provides it through several product - Replication Server, the IBM Data Server Client, and all editions of DB2 and InfoSphere Warehouse.*
Long-time CDC users may ask what's happening to CDC's differential refresh and why they would want to use asntdiff instead of differential refresh. First understand that differential refresh is alive and well and it's not going anywhere :) asntdiff is just an option available to you.
To understand when you might want to use asntdiff, understand the basics of how it works.
- asntdiff accepts two queries as input and compares the result sets.
- You can use almost any query you can write against source and target tables.
So, the first reason to consider asntdiff is times when differential refresh's restrictions could be overcome by writing queries to get the result sets you need. For example, asntdiff may be an alternative if one of the following differential refresh restrictions applies to your replication configuration:
- Differential refresh is only available for tables that use Standard replication.
- Derived columns in the source table are not supported.
- Target columns are ignored if they are mapped to derived expressions, constants, or journal control fields.
- Key columns of the target table must be mapped directly to columns in the source table.
Next, asntdiff is independent of data replication and can be started from a command line. Among other things, this means:
- It can made part of a z/OS batch job and scheduled.
- It can be used while a CDC subscription is running
One major point to be aware of with asntdiff is how it works with heterogeneous data. For example, when you want to compare data being replicated from Oracle to DB2. asntdiff was originally written for DB2 databases. As a result, it requires IBM data federation technology to query databases such as Oracle. The good news is that InfoSphere Data Replication provides data federation for use with data replication configurations.
If you're not familiar with asntdiff and want to give it a try, see the ChannelDB2.com blog post titled Compare the Rows of Two Tables
. If you have questions, feel free to post them in the CDC message board here on developerWorks.
* Yes, technically, you could already use asntdiff with CDC on UNIX or Window since it comes in so many IBM products on UNIX and Windows. However, if you wanted to use it on z/OS, you could only get it through Replication Server. It's now in InfoSphere Data Replication as well.
With a mere 4 weeks until IBM's 2013 Information on Demand, the data replication team thought it might be helpful to have a complete listing of all data replication sessions at IOD. From client presentations and our product roadmap to sneak peeks at new IBM Data Replication functionality, our sessions run the gamut!
Simply take a gander at the sessions below then go to the IOD agenda builder, click on Create Sign In, and then enter your confirmation number and the email address that you used to register for the conference. Create your agenda today!
I've added three new videos to my channel. They walk through configuring, operating and monitoring data replication using the CDC Management Console. This is basically the same thing you'd get if you came by the InfoSphere demo room at Information On Demand (now Insight) and agreed to let me show you a quick demo of CDC.
Here's the link to my channel "James talks about Data Replication":
If you're looking for an excellent way to replicate changed data from a wide range of databases into a Netezza appliance, you can do so through InfoSphere Data Replication
. The latest release provides an Apply program that is both native to Netezza and optimized for Netezza targets. This Apply is built from Data Replication's CDC technology and is also compatible with the CDC technology found in InfoSphere Change Data Capture and InfoSphere Classic Change Data Capture for z/OS
. This means you can replicate data to Netezza from source databases ranging from Oracle, DB2, and others on UNIX or Windows to DB2* and IMS on the mainframe. Ordering information can be found in the Data Replication announcement letter on ibm.com
* Data Replication's CDC Apply program cannot be used to feed changed data to the IBM DB2 Analytics Accelerator (IDAA).
With the announcement of InfoSphere Data Replication 10.1.2
, IBM added a product called InfoSphere Data Replication for Database Migration. This new product is a tool to help you with hardware and database upgrades. It is intended for short-term use. For example, if you're upgrading to a totally new hardware platform, the new Data Replication product keeps two copies of your database in sync - one copy on your old hardware and the other on your new hardware. This gives you the time you need - a few weeks or several months - to migrate and test applications before you turn off the old hardware. A similar scenario is possible if you're just upgrading databases or if you're upgrading hardware and databases simultaneously.
The first release of the new product is available for three different combinations of source and target databases (three different from and to combinations):
- Oracle to Oracle
- Oracle to DB2 for Linux, UNIX, and Windows (LUW)
- DB2 LUW to DB2 LUW
It provides only the data replication function needed for database migration. Specifically:
- Unidirectional (one-way) replication
- If you need multi-way replication, you need to buy the full Data Replication product.
- One source and target database pair.
- In other words, a single copy of the product can only be replicating between two databases at any given time.
- However, after you finish migrating a given database, you can move the product to another system and migrate another database.
- Data transformations when the source is an Oracle database and the target is DB2 LUW.
- If you need transformations for other source and target combinations, you need to buy the full Data Replication product.
- Replication of add and remove table partitions when both source and target are Oracle databases.
- If you need other DDL replication, you need to buy the full Data Replication product.
Of course, like the full Data Replication product, the new product contains all IBM Data Replication technologies - CDC, Q Replication, and SQL Replication. However, for those of you familiar with the older products- InfoSphere CDC and InfoSphere Replication Server - there is no database migration edition of those two products.
Note that there are two licensing differences for this Data Replication product when compared to many other products. First, this one is licensed by target server install instead of a PVU (processor value unit) count. That means for each target install you license, you can install at a single source for no additional charge. Second, IBM does not offer a non-production version. Therefore, you buy the same product for both production and non-production uses. This isn't bad since this database migration product is significantly cheaper than the full Data Replication product. To verify these licensing points and others, always see the the license file on ibm.com
as the official word in how licensing works.