Number of Instances Required
You can have multiple instances of InfoSphere CDC running on the same server, each would have its own copy of storage, metadata, etc.
A separate InfoSphere CDC instance is required for each database that you want to replicate from, except in the case where a single instance is being used as both a source and a target
Additional instances may be required for the following reasons:
If you hit the maximum number of subscriptions for a single instance
If you have extremely large log volume and you need to split the source into multiple instances. For further information on this situation, please refer to the performance tuning presentation
Log retention policies
For InfoSphere CDC LUW, utilize dmshowlogdependency command to develop your retention procedures. This command will tell you when InfoSphere CDC has completed with a log
For InfoSphere CDC i, utilize the CHGJRNDM command to manage journal receivers
For InfoSphere CDC z, there is no command available. Generally not a requirement as most z shops keep logs around for 10 days. If required, you can utilize the earliest open position indicated in the event log when InfoSphere CDC z starts replication
You need to consider and accommodate for cases when replication will be down for a period of time
Rule of Thumb:
Successful implementations typically have 5+ days of logs retained
If you do not have sufficient log retention, you need to be prepared to do table refreshes if something unexpected happens in your environment
Modified by GlenSakuth
I have had many requests to share best practices when using IBM InfoSphere Change Data Capture (from this point forward in the blog referred to as CDC). I will try to add new tips and techniques on a regular basis.
Along with many of the best practices posts, I will include items denoted by "Rule of Thumb". These are general guidelines that will help in your planning. I will endeavor to provide reasons or context for the guidance. The Rules of Thumb should not be treated as hard limits, rather as useful guidance. If your needs fall significantly outside the guidance, it certainly does not mean that it can not be done. Rather, it would be best to engage with an InfoSphere CDC subject matter expert, and potentially you may want to consider IBM Services for assistance.
Steady State Operations
This post is the answer to one of the FAQs found in License Tips for IBM Data Replication.
You may have seen a recent announcement on ibm.com
that says IBM would no longer be marketing it's older data replication products in 2013. That includes InfoSphere CDC. Why?
And what happens to the CDC technology?
Over the years, IBM provided its data replication technologies
through a lot of different products. For example, IBM used to offer two major
data replication products at the same time -
InfoSphere CDC and InfoSphere Replication Server. That was a little
even to some IBM people. To simplify the situation, IBM consolidated all it's replication technologies into a single product called IBM InfoSphere Data Replication
(IIDR). Once IIDR was available, the older products no longer needed to be sold to new customers. That's why the end of marketing was announced. However, the replication technologies - CDC, Q Replication, and SQL Replication - are still alive and well. You can
continue to use them as you always
have. Of course, you may have two related questions:
- Are the older products still being supported?
- How do you move from your old InfoSphere CDC product to IIDR?
If you have any questions, feel free to post them in the comments section of this blog.
The IBM Redbook titled "Smarter Business: Dynamic Information with IBM InfoSphere Data Replication CDC" is now available and can be found: http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247941.html?Open
This Redbook covers a wide range of topics from InfoSphere CDC use cases, solution topologies, features and functionality, performance, environmental considerations and automation. This is a great source of information if you are wondering how best to set up InfoSphere CDC, how do you fit it into a resilient environment, etc.
With the announcement of InfoSphere Data Replication 10.1.2
, IBM added a product called InfoSphere Data Replication for Database Migration. This new product is a tool to help you with hardware and database upgrades. It is intended for short-term use. For example, if you're upgrading to a totally new hardware platform, the new Data Replication product keeps two copies of your database in sync - one copy on your old hardware and the other on your new hardware. This gives you the time you need - a few weeks or several months - to migrate and test applications before you turn off the old hardware. A similar scenario is possible if you're just upgrading databases or if you're upgrading hardware and databases simultaneously.
The first release of the new product is available for three different combinations of source and target databases (three different from and to combinations):
- Oracle to Oracle
- Oracle to DB2 for Linux, UNIX, and Windows (LUW)
- DB2 LUW to DB2 LUW
It provides only the data replication function needed for database migration. Specifically:
- Unidirectional (one-way) replication
- If you need multi-way replication, you need to buy the full Data Replication product.
- One source and target database pair.
- In other words, a single copy of the product can only be replicating between two databases at any given time.
- However, after you finish migrating a given database, you can move the product to another system and migrate another database.
- Data transformations when the source is an Oracle database and the target is DB2 LUW.
- If you need transformations for other source and target combinations, you need to buy the full Data Replication product.
- Replication of add and remove table partitions when both source and target are Oracle databases.
- If you need other DDL replication, you need to buy the full Data Replication product.
Of course, like the full Data Replication product, the new product contains all IBM Data Replication technologies - CDC, Q Replication, and SQL Replication. However, for those of you familiar with the older products- InfoSphere CDC and InfoSphere Replication Server - there is no database migration edition of those two products.
Note that there are two licensing differences for this Data Replication product when compared to many other products. First, this one is licensed by target server install instead of a PVU (processor value unit) count. That means for each target install you license, you can install at a single source for no additional charge. Second, IBM does not offer a non-production version. Therefore, you buy the same product for both production and non-production uses. This isn't bad since this database migration product is significantly cheaper than the full Data Replication product. To verify these licensing points and others, always see the the license file on ibm.com
as the official word in how licensing works.
If you're looking for an excellent way to replicate changed data from a wide range of databases into a Netezza appliance, you can do so through InfoSphere Data Replication
. The latest release provides an Apply program that is both native to Netezza and optimized for Netezza targets. This Apply is built from Data Replication's CDC technology and is also compatible with the CDC technology found in InfoSphere Change Data Capture and InfoSphere Classic Change Data Capture for z/OS
. This means you can replicate data to Netezza from source databases ranging from Oracle, DB2, and others on UNIX or Windows to DB2* and IMS on the mainframe. Ordering information can be found in the Data Replication announcement letter on ibm.com
* Data Replication's CDC Apply program cannot be used to feed changed data to the IBM DB2 Analytics Accelerator (IDAA).
Now that IBM has packaged its major data replication technologies into a single product, InfoSphere Data Replication
, a lot of people are asking what they can take advantage of that they couldn't with the older products (InfoSphere CDC and InfoSphere Replication Server). Other than the obvious point of having access to multiple technologies, you can now use IBM's table compare utility, asntdiff
, with CDC. asntdiff is a general-purpose utility that compares the data from two queries. IBM provides it through several product - Replication Server, the IBM Data Server Client, and all editions of DB2 and InfoSphere Warehouse.*
Long-time CDC users may ask what's happening to CDC's differential refresh and why they would want to use asntdiff instead of differential refresh. First understand that differential refresh is alive and well and it's not going anywhere :) asntdiff is just an option available to you.
To understand when you might want to use asntdiff, understand the basics of how it works.
- asntdiff accepts two queries as input and compares the result sets.
- You can use almost any query you can write against source and target tables.
So, the first reason to consider asntdiff is times when differential refresh's restrictions could be overcome by writing queries to get the result sets you need. For example, asntdiff may be an alternative if one of the following differential refresh restrictions applies to your replication configuration:
- Differential refresh is only available for tables that use Standard replication.
- Derived columns in the source table are not supported.
- Target columns are ignored if they are mapped to derived expressions, constants, or journal control fields.
- Key columns of the target table must be mapped directly to columns in the source table.
Next, asntdiff is independent of data replication and can be started from a command line. Among other things, this means:
- It can made part of a z/OS batch job and scheduled.
- It can be used while a CDC subscription is running
One major point to be aware of with asntdiff is how it works with heterogeneous data. For example, when you want to compare data being replicated from Oracle to DB2. asntdiff was originally written for DB2 databases. As a result, it requires IBM data federation technology to query databases such as Oracle. The good news is that InfoSphere Data Replication provides data federation for use with data replication configurations.
If you're not familiar with asntdiff and want to give it a try, see the ChannelDB2.com blog post titled Compare the Rows of Two Tables
. If you have questions, feel free to post them in the CDC message board here on developerWorks.
* Yes, technically, you could already use asntdiff with CDC on UNIX or Window since it comes in so many IBM products on UNIX and Windows. However, if you wanted to use it on z/OS, you could only get it through Replication Server. It's now in InfoSphere Data Replication as well.
In 2011, IBM released three new data replication products:
One question that comes up is whether the two IMS replication products are compatible with either the new Data Replication product or the existing InfoSphere CDC products. The answer is yes - the IMS products are compatible with both new and existing products that contain the CDC technology. IMore specifically, they can provide IMS changed data to any data replication solution that you can build with IBM's CDC technology. For example, you can create unidirectional (one-way) subscriptions that feed IMS changed data to any database that can be targeted by CDC:
Two notes about this picture:
- IBM recommends you use the CDC technology in IIDR if you do not own InfoSphere CDC.
- The target DB2 can be DB2 for z/OS, DB2 LUW, or DB2 for System i.
You could also feed IMS changed data into other business software such as ETL, IBM's DataStage, and ESBs:
In other words, the new IMS data replication products extend the reach of IBM's CDC technology by adding IMS as a source for log-based capture of changed data. If you have technical questions, see the Classic CDC section of the Information Center
Over the next month or so, we'll be updating the look of the CDC Forum and adding content. For example, you may have noticed we added a data replication icon and a wiki widget. More is in store. If you have requests, feel free to post a comment to this blog post. No guarantees, but we'll do what we can :)