IBM Data Replication's CDC News

There are multiple deployment models available for InfoSphere CDC.   The deployment model chosen for the source system will significantly affect the complexity of implementation.

Here are the CDC source deployment options from the least complex to the most complex:

1. InfoSphere CDC scraper runs on the source database server

2. InfoSphere CDC scraper runs on a remote tier reading logs from a shared disk (SAN)

• This configuration is available for Oracle and Sybase.  DB2 LUW has a similar capability, but utilizes a remote client instead of reading from a SAN.

3. InfoSphere CDC scraper runs on a remote tier using log shipping

• This configuration is only available for Oracle.

Rule of Thumb

You should always use the least complex deployment option that will meet the business needs.  The vast majority of CDC users install InfoSphere CDC on the source database server.

With a mere 4 weeks until IBM's 2013 Information on Demand, the data replication team thought it might be helpful to have a complete listing of all data replication sessions at IOD. From client presentations and our product roadmap to sneak peeks at new IBM Data Replication functionality, our sessions run the gamut!

Simply take a gander at the sessions below then go to the IOD agenda builder, click on Create Sign In, and then enter your confirmation number and the email address that you used to register for the conference.  Create your agenda today!

If you need to understand more about these examples, we'll have pictures and add a few more examples in a future post that talks about when you need to buy CDC.

Of course, the last question is - can I still build DB2 DR, HA, and Active-Active solutions using the Q Replication built into the DB2 Advanced Editions?  Yes, absolutely.  The addition of CDC to DB2 does not change this.

----------------

* Multiple DB2 instances can be created from a single DB2 install.  Each instance can use the bundled CDC to replicate up to the entitled number of backup instances.

Single Scrape

When multiple subscriptions are running in a single instance, it is usually advantageous to utilize a single scrape mechanism.  If you don't use single scrape, and you have 'n' subscriptions, CDC would read the log 'n' times.  If you utilize single scrape, CDC will only read the log once which will utilize fewer system resources.

• On by default for InfoSphere CDC LUW
• You must configure the log cache for InfoSphere CDC z
• Not available on InfoSphere CDC i or CDC Informix

You need to size the single scrape cache appropriately for optimal performance:

• If the cache is too small the following will occur:
  • LUW – A private scraper will be launched which will consume additional resources
    •  Set staging_store_disk_quota_gb system parameter appropriately to avoid
  • Z - With the log cache, each subscription attempts to read its data from the cache – it will read directly from the IFI if the data is no longer available from the cache
    • Use the following to configure CACHELEVEL1SIZE, CACHEBLOCKSIZE, CACHELEVEL1RESERVED

Number of Tables in a Subscription

Rule of Thumb

• This is certainly not a hard limit, but in general it is best to keep the number of tables in a subscription under 1000

Considerations for the number of tables include:

• With too many tables (over 1000) in a subscription, loading and managing the tables in the Management Console GUI will be slow
  • This may not be a consideration if you are controlling your replication via scripting/automation
• If the number of tables exceed 1000 then promotion in the management console will take a significant amount of time, and additional memory would need to be allocated
• From an engine perspective:
  • With CDC LUW if you want to go beyond 1000 tables you need to increase the memory allocated to the InfoSphere CDC Instance
  • CDC i can accommodate well over 2000 tables in a subscription
  • CDC z can accommodate well over 1000 tables in a subscription
    • Note, the number can be significantly higher, but there are implications to the number of subscriptions you have due to limits on below the bar memory

Number of CDC Subscriptions Required

A Subscription is a logical container that describes the replication configuration for tables from a source to a target datastore.  Once the subscription is created, you create table mappings within the subscription for the group of tables you wish to replicate

An important part of planning an InfoSphere CDC implementation is to choose the appropriate number of subscriptions to meet your requirements
 

Rule of Thumb:

• Starting with the minimum number of subscriptions and only increasing due to valid reasons, is the optimal approach
  • This will ensure efficient use of resources as well as require a lower level of maintenance

It may require an iterative process before you have a good balance

• The number of subscriptions will impact the resource utilization of the server (more CPU and RAM are needed) and performance of InfoSphere CDC
• Note that tables with referential integrity or ones where the data must be synchronized at all times must reside in the same subscription since different subscriptions may be at different points in the log
• The following are valid reasons to increase the number of subscriptions:
  • Requirement to replicate one source table to multiple targets
  • You need to increase the number of applies once it has been determined that it is the apply that is affecting the performance and you want further parallelism
  • Management of replication for groups of tables, in cases where some tables only require mirroring with a scheduled end time, while others require continuous or they are active at different times of the day
  • You have too many tables in a single subscription which is affecting start-up performance
  • You have multiple independent business applications that you need to mirror, but want to be able to deal with maintenance independently

Number of Subscriptions per CDC Instance

For best resource utilization, and easiest management, you want to keep the number of CDC Instances and Subscriptions to the minimum. 

Rule of Thumb:

• InfoSphere CDC LUW can generally accommodate up to 50 subscriptions per instance (either source or target)
• InfoSphere CDC z can generally accommodate up to 20 combined source and target subscriptions per instance and a hard maximum of 50 subscriptions per instance
  • Note: For CDC z if you have three or more source subscriptions in an instance, for optimal resource utilization, you need to ensure that the log cache is configured
• InfoSphere CDC i can generally accommodate up to 25 source subscriptions per instance, and 25 subscriptions in a target instance
  • Note that InfoSphere CDC i does not have the single scrape feature, so each additional subscription will require proportionally extra CPU resource if reading from a single journal.  Thus, if you have multiple subscriptions you will achieve better efficiency if separate journals can be used for each subscription

Setting Up Notifications (Sometimes referred to as Alerts and Alarms)

There are various means of checking and understanding replication status, performance, etc.  One important aspect is to be able to be notified in the event of a replication issue be it an error, or latency.  Notifications can be sent for any event message that InfoSphere CDC produces.

• Appropriate notifications settings will alert InfoSphere CDC administrators of issues with the environment in a timely manner so they can be addressed
• Notifications can be setup for various categories on the source and target and at the datastore or individual subscription level
• Messages can also be filtered based on severity: Status, informational, operational, error and fatal
• Latency notifications can be setup to monitor performance issues at a subscription level.  A message can be sent to the event log when a warning threshold is passed, and another message if an error threshold is passed
• InfoSphere CDC for z/OS also allows users to select specific messages to be directed to the console – see CONSOLEMSGS keyword

Notification can be directed to platform specific destinations or a custom user exit program

• LUW
• E-mail
• SMTP
• Specify e-mail address and password
• Unix System log
• Custom Java User Exit Program
•  

z/OS

• CHCPRINT spool file
• SYSLOG
• User Exit

IBM i

• Message Queue
• User Exit

Rule of Thumb:

• The general practice is to have notifications set up for all Fatal and Error messages (events), as well as to have a notification for a latency threshold

Number of Instances Required

• You can have multiple instances of InfoSphere CDC running on the same server, each would have its own copy of storage, metadata, etc.
• A separate InfoSphere CDC instance is required for each database that you want to replicate from, except in the case where a single instance is being used as both a source and a target

• Additional instances may be required for the following reasons:
  • If you hit the maximum number of subscriptions for a single instance
  • If you have extremely large log volume and you need to split the source into multiple instances.  For further information on this situation, please refer to the performance tuning presentation

Log retention policies

• For InfoSphere CDC LUW, utilize dmshowlogdependency command to develop your retention procedures.  This command will tell you when InfoSphere CDC has completed with a log
• For InfoSphere CDC i, utilize the CHGJRNDM command to manage journal receivers
• For InfoSphere CDC z, there is no command available.  Generally not a requirement as most z shops keep logs around for 10 days.  If required, you can utilize the earliest open position indicated in the event log when InfoSphere CDC z starts replication

• You need to consider and accommodate for cases when replication will be down for a period of time

Rule of Thumb:

• Successful implementations typically have 5+ days of logs retained
• If you do not have sufficient log retention, you need to be prepared to do table refreshes if something unexpected happens in your environment

I have had many requests to share best practices when using IBM InfoSphere Change Data Capture (from this point forward in the blog referred to as CDC).  I will try to add new tips and techniques on a regular basis. 

Along with many of the best practices posts, I will include items denoted by "Rule of Thumb".  These are general guidelines that will help in your planning.  I will endeavor to provide reasons or context for the guidance.  The Rules of Thumb should not be treated as hard limits, rather as useful guidance.  If your needs fall significantly outside the guidance, it certainly does not mean that it can not be done.  Rather, it would be best to engage with an InfoSphere CDC subject matter expert, and potentially you may want to consider IBM Services for assistance.

Planning Deployments

• Best Practice - Logging Requirements
• Best practice - Log Retention Policies
• Best Practice - Required CPU on Source
• Best Practice - Memory Requirements on LUW
• Best Practice - Target Considerations

Deploying

• Best practice - Number of CDC instances Required
• Best practice - Number of Subscriptions per CDC Instance
• Best practice - Number of CDC Subscriptions Required
• Best Practice - Number of Tables in a Subscription
• Best Practice - Single Scrape
• Best Practice - Long Running & Large Transactions

Steady State Operations

• Best practice - Setting up Notifications (Alerts & Alarms)
• Best Practice - Things to know that you may not be aware of