IBM Data Replication's CDC News

I have had many requests to share best practices when using IBM InfoSphere Change Data Capture (from this point forward in the blog referred to as CDC).  I will try to add new tips and techniques on a regular basis. 

Along with many of the best practices posts, I will include items denoted by "Rule of Thumb".  These are general guidelines that will help in your planning.  I will endeavor to provide reasons or context for the guidance.  The Rules of Thumb should not be treated as hard limits, rather as useful guidance.  If your needs fall significantly outside the guidance, it certainly does not mean that it can not be done.  Rather, it would be best to engage with an InfoSphere CDC subject matter expert, and potentially you may want to consider IBM Services for assistance.

For a comprehensive list of best practices, please see the parent community main page:

https://www.ibm.com/developerworks/community/groups/service/html/communityoverview?communityUuid=a9b542e4-7c66-4cf3-8f7b-8a37a4fdef0c

Planning Deployments

• Best Practice - Logging Requirements
• Best practice - Log Retention Policies
• Best Practice - Required CPU on Source
• Best Practice - Memory Requirements on LUW
• Best Practice - Target Considerations

Deploying

• Best practice - Number of CDC instances Required
• Best practice - Number of Subscriptions per CDC Instance
• Best practice - Number of CDC Subscriptions Required
• Best Practice - Number of Tables in a Subscription
• Best Practice - Single Scrape
• Best Practice - Long Running & Large Transactions

Steady State Operations

• Best practice - Setting up Notifications (Alerts & Alarms)
• Best Practice - Things to know that you may not be aware of

Log retention policies

• For InfoSphere CDC LUW, utilize dmshowlogdependency command to develop your retention procedures.  This command will tell you when InfoSphere CDC has completed with a log
• For InfoSphere CDC i, utilize the CHGJRNDM command to manage journal receivers
• For InfoSphere CDC z, there is no command available.  Generally not a requirement as most z shops keep logs around for 10 days.  If required, you can utilize the earliest open position indicated in the event log when InfoSphere CDC z starts replication

• You need to consider and accommodate for cases when replication will be down for a period of time

Rule of Thumb:

• Successful implementations typically have 5+ days of logs retained
• If you do not have sufficient log retention, you need to be prepared to do table refreshes if something unexpected happens in your environment

References:

search results for dmshowlogdepencency

http://www.ibm.com/support/knowledgecenter/search/dmshowlogdependency?scope=SSTRGZ_11.3.3

Oracle dmshowlogdepencency

http://www.ibm.com/support/knowledgecenter/SSTRGZ_11.3.3/com.ibm.cdcdoc.cdcfororacle.doc/refs/dmshowlogdependency.html

SQL Server dmshowlogdepencency

http://www.ibm.com/support/knowledgecenter/SSTRGZ_11.3.3/com.ibm.cdcdoc.cdcformssql.doc/concepts/understandinghowcdcinteractswithyourdatabase.html

DB2 (LUW) dmshowlogdepencency

http://www.ibm.com/support/knowledgecenter/SSTRGZ_11.3.3/com.ibm.cdcdoc.cdcfordb2luw.doc/refs/dmshowlogdependency.html

Number of Instances Required

• You can have multiple instances of InfoSphere CDC running on the same server, each would have its own copy of storage, metadata, etc.
• A separate InfoSphere CDC instance is required for each database that you want to replicate from, except in the case where a single instance is being used as both a source and a target

• Additional instances may be required for the following reasons:
  • If you hit the maximum number of subscriptions for a single instance
  • If you have extremely large log volume and you need to split the source into multiple instances.  For further information on this situation, please refer to the performance tuning presentation

Setting Up Notifications (Sometimes referred to as Alerts and Alarms)

There are various means of checking and understanding replication status, performance, etc.  One important aspect is to be able to be notified in the event of a replication issue be it an error, or latency.  Notifications can be sent for any event message that InfoSphere CDC produces.

• Appropriate notifications settings will alert InfoSphere CDC administrators of issues with the environment in a timely manner so they can be addressed
• Notifications can be setup for various categories on the source and target and at the datastore or individual subscription level
• Messages can also be filtered based on severity: Status, informational, operational, error and fatal
• Latency notifications can be setup to monitor performance issues at a subscription level.  A message can be sent to the event log when a warning threshold is passed, and another message if an error threshold is passed
• InfoSphere CDC for z/OS also allows users to select specific messages to be directed to the console – see CONSOLEMSGS keyword

Notification can be directed to platform specific destinations or a custom user exit program

• LUW
• E-mail
• SMTP
• Specify e-mail address and password
• Unix System log
• Custom Java User Exit Program
•  

z/OS

• CHCPRINT spool file
• SYSLOG
• User Exit

IBM i

• Message Queue
• User Exit

Rule of Thumb:

• The general practice is to have notifications set up for all Fatal and Error messages (events), as well as to have a notification for a latency threshold

Number of Subscriptions per CDC Instance

For best resource utilization, and easiest management, you want to keep the number of CDC Instances and Subscriptions to the minimum. 

Rule of Thumb:

• InfoSphere CDC LUW can generally accommodate up to 50 subscriptions per instance (either source or target)
• InfoSphere CDC z can generally accommodate up to 20 combined source and target subscriptions per instance and a hard maximum of 50 subscriptions per instance
  • Note: For CDC z if you have three or more source subscriptions in an instance, for optimal resource utilization, you need to ensure that the log cache is configured
• InfoSphere CDC i can generally accommodate up to 25 source subscriptions per instance, and 25 subscriptions in a target instance
  • Note that InfoSphere CDC i does not have the single scrape feature, so each additional subscription will require proportionally extra CPU resource if reading from a single journal.  Thus, if you have multiple subscriptions you will achieve better efficiency if separate journals can be used for each subscription

Number of CDC Subscriptions Required

A Subscription is a logical container that describes the replication configuration for tables from a source to a target datastore.  Once the subscription is created, you create table mappings within the subscription for the group of tables you wish to replicate

An important part of planning an InfoSphere CDC implementation is to choose the appropriate number of subscriptions to meet your requirements
 

More information can be found in the CDC performance documents :

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W8d78486eafb9_4a06_a482_7e7962f5ac59/page/IIDR%20Wiki?section=CDC%20Performance

For a comprehensive list of best practices, please see the parent community main page:

https://www.ibm.com/developerworks/community/groups/service/html/communityoverview?communityUuid=a9b542e4-7c66-4cf3-8f7b-8a37a4fdef0c

Rule of Thumb:

• Starting with the minimum number of subscriptions and only increasing due to valid reasons, is the optimal approach
  • This will ensure efficient use of resources as well as require a lower level of maintenance

It may require an iterative process before you have a good balance

• The number of subscriptions will impact the resource utilization of the server (more CPU and RAM are needed) and performance of InfoSphere CDC
• Note that tables with referential integrity or ones where the data must be synchronized at all times must reside in the same subscription since different subscriptions may be at different points in the log
• The following are valid reasons to increase the number of subscriptions:
  • Requirement to replicate one source table to multiple targets
  • You need to increase the number of applies once it has been determined that it is the apply that is affecting the performance and you want further parallelism
  • Management of replication for groups of tables, in cases where some tables only require mirroring with a scheduled end time, while others require continuous or they are active at different times of the day
  • You have too many tables in a single subscription which is affecting start-up performance
  • You have multiple independent business applications that you need to mirror, but want to be able to deal with maintenance independently

Number of Tables in a Subscription

Rule of Thumb

• This is certainly not a hard limit, but in general it is best to keep the number of tables in a subscription under 1000

Considerations for the number of tables include:

• With too many tables (over 1000) in a subscription, loading and managing the tables in the Management Console GUI will be slow
  • This may not be a consideration if you are controlling your replication via scripting/automation
• If the number of tables exceed 1000 then promotion in the management console will take a significant amount of time, and additional memory would need to be allocated
• From an engine perspective:
  • With CDC LUW if you want to go beyond 1000 tables you need to increase the memory allocated to the InfoSphere CDC Instance
    • If the target is flatfile or HDFS, then an upper limit on the number of tables in the subscription is 800.  Additionally, you would need to allocate some additional memory if you have more than a couple hundred tables.
  • CDC i can accommodate well over 2000 tables in a subscription
  • CDC z can accommodate well over 1000 tables in a subscription
    • Note, the number can be significantly higher, but there are implications to the number of subscriptions you have due to limits on below the bar memory

Shared Scrape (sometimes referred to as Single Scrape)

When multiple subscriptions are running in a single instance, it is usually advantageous to utilize a shared scrape mechanism.  If you don't use a shared scrape, and you have 'n' subscriptions, CDC would read the log 'n' times.  If you utilize shared scrape, CDC will only read the log once which will utilize fewer system resources.

• On by default for InfoSphere CDC LUW
• You must configure the log cache for InfoSphere CDC z
• Not available on InfoSphere CDC i or CDC Informix

You need to size the shared scrape cache appropriately for optimal performance:

• If the cache is too small the following will occur:
  • LUW – A private scraper will be launched which will consume additional resources
    •  Set staging_store_disk_quota_gb system parameter appropriately to avoid
  • Z - With the log cache, each subscription attempts to read its data from the cache – it will read directly from the IFI if the data is no longer available from the cache
    • Use the following to configure CACHELEVEL1SIZE, CACHEBLOCKSIZE, CACHELEVEL1RESERVED

If you need to understand more about these examples, we'll have pictures and add a few more examples in a future post that talks about when you need to buy CDC.

Of course, the last question is - can I still build DB2 DR, HA, and Active-Active solutions using the Q Replication built into the DB2 Advanced Editions?  Yes, absolutely.  The addition of CDC to DB2 does not change this.

----------------

* Multiple DB2 instances can be created from a single DB2 install.  Each instance can use the bundled CDC to replicate up to the entitled number of backup instances.

With a mere 4 weeks until IBM's 2013 Information on Demand, the data replication team thought it might be helpful to have a complete listing of all data replication sessions at IOD. From client presentations and our product roadmap to sneak peeks at new IBM Data Replication functionality, our sessions run the gamut!

Simply take a gander at the sessions below then go to the IOD agenda builder, click on Create Sign In, and then enter your confirmation number and the email address that you used to register for the conference.  Create your agenda today!

There are multiple deployment models available for InfoSphere CDC.   The deployment model chosen for the source system will significantly affect the complexity of implementation.

Here are the CDC source deployment options from the least complex to the most complex:

1. InfoSphere CDC scraper runs on the source database server

2. InfoSphere CDC scraper runs on a remote tier reading logs from a shared disk (SAN)

• This configuration is available for Oracle and Sybase.  DB2 LUW has a similar capability, but utilizes a remote client instead of reading from a SAN.

3. InfoSphere CDC scraper runs on a remote tier using log shipping

• This configuration is only available for Oracle.

Rule of Thumb

You should always use the least complex deployment option that will meet the business needs.  The vast majority of CDC users install InfoSphere CDC on the source database server.

There are many deployment models available for InfoSphere Data Replication's CDC technology of which DataStage integration is a popular one.  The deployment option selected will significantly affect the complexity, performance, and reliability of the implementation.   If possible, the best solution is always to use CDC direct replication (i.e. do not add DataStage to the mix). 

CDC integration with DataStage is the right solution for replication when:

• You need to target a database that CDC doesn't directly support and is not appropriate for CDC FlexRep
• Complex transformations are required that could not be handled natively with CDC, such as complex table look-ups
• When integrating with MDM

Cons of replicating from CDC to DataStage to an eventual target database:

• Performance going through DataStage (no matter which integration option is chosen) will be significantly slower than applying via a CDC target directly to the database
  • The exception to this rule is when targeting Teradata, if you use DataStage flatfile integration, the throughput will be higher than CDC direct to Teradata
• Adding DataStage into the replication stream introduces additional points of failure
• Having a resilient CDC installation is more complex if DataStage is also involved
• When integrating with DataStage, there are two independent GUIs for configuration, and two places required to monitor the replication stream
• There is significant development effort developing DataStage jobs for each additional table added to replication
• Incorrect DataStage job design can negatively affect transactional integrity and cause data corruption
• The maximum number of tables per CDC subscription is lower if targeting DataStage
• The CDC External Refresh does not work when targeting DataStage.  A separate process would have to be put in place to de-dup duplicate records produced during the "in-doubt" period of a refresh (the captured changes that occurred while the source date was being refreshed).

Link to Wiki containing best practices for integration with DataStage

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W8d78486eafb9_4a06_a482_7e7962f5ac59/page/IIDR%20Wiki?section=DataStage

Blog Author: Davendra Paltoo, Offering Manager,Data Replication

With growing volumes, variety, and velocity of data, the challenge of protecting data continues. Every organization today is striving to protect its customer data and other data as the cost of data breaches are high. The 2017 Ponemon Cost of Data Breach Study, reports that the global average cost of a data breach is $3.62 million. The average cost for each lost or stolen record containing sensitive and confidential information decreased from $158 in 2016 to $141 in 2017. Despite the decline in the individual cost per record, companies report having larger breaches in 2017. The average size of the data breaches reported in this research increased 1.8 percent to more than 24,000 records per incident.

Security professionals are shifting their focus from device-specific controls to a data-centric approach that focuses on securing the apps and data and controlling access. Business, security, and privacy leaders understand that industry standard security practices have to be adopted to protect an organization’s data.

One of the reasons for security of data being compromised is when industry standard authentication mechanisms are not applied.

As part of movement to more centralized governance models for ease of administration and better security, organizations commonly want to centrally manage user credentials, security policies and access rights as part of managing access to their applications and data.

As a result, many organizations manage their user credentials, security policies and access rights in a central repository by implementing a Lightweight Directory Access Protocol (LDAP) compliant Directory Service such as IBM’s Tivoli, Microsoft’s Active Directory, and Apache’s Directory Services.

In addition, organizations also prefer business software to leverage these directory services rather than use decentralized, individually managed user credentials, security policies or access rights that could potentially be created for each piece of software deployed.

To help cater to the aforementioned security needs of today’s digital businesses, IBM Data Replication’s Change Data Capture (“CDC”) technology has introduced support for integration with LDAP directory services. Traditionally, the CDC Access Server authenticates users, stores user credentials and data access information, and acts as the centralized communicator between all replication agents and Management Console clients.

Now, starting with the IIDR 11.4.0.0-10291 Management Console and Access Server delivery, users can choose to have an LDAP server manage their CDC user credentials, user authentication, and data store access information to help users conform to LDAP based centralized security architecture in their enterprise.

For more information about the new IIDR (CDC) LDAP enablement and for details on how to configure LDAP with IIDR (CDC) please refer to the below links.

• https://www.ibm.com/developerworks/community/blogs/a9b542e4-7c66-4cf3-8f7b-8a37a4fdef0c/entry/IIDR_11_4_0_0_10288_Management_Console_and_Access_Server?lang=es
• https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.4.0/com.ibm.cdcdoc.installingasandmc.doc/concepts/ldapovu.html

Blog Post by : Davendra Paltoo, Offering Manager, Data Replication

Follow Davendra on Twitter at : https://twitter.com/Davendr18397388

Real time analytics can provide real time insights. When businesses have data at the right time, they can be more efficient and make the right tactical and strategic decisions. Apache Kafka® is used for building Data hubs or landing zones, building real-time data pipelines, and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

The Apache Kafka (https://kafka.apache.org/) platform now has a vibrant associated eco-system.

When messages are landed in a Kafka cluster, there are a variety of available connectors or consumers that can in turn retrieve messages and deliver such messages to target destinations such as HDFS and Amazon S3. Users can save time and costs by using one of these available consumers, such as those listed on https://docs.confluent.io/current/connect/connectors.html or https://community.hortonworks.com/topics/Kafka.html

IIDR (CDC), with the initial version of its 11.4.0 release, provided users the ability to replicate from any supported CDC Replication source to a Kafka cluster by using the IIDR (CDC) target Replication Engine for Kafka. This engine writes Kafka messages that contain the replicated data to Kafka topics. The replicated data in the Kafka messages is by default written in the Avro binary format. Consumers that want to read these messages from Kafka clusters needed to utilize an Avro binary deserializer.

Users on the other hand, desired more flexibility in the IIDR (CDC) Kafka apply so that it could help users produce the various permutations of data formats of messages written to Kafka expected by the wide variety of off the shelf, custom connectors and consumer applications.

To help customers solve this challenge, IIDR (CDC) has recently introduced support for "Kafka custom operation processors" (KCOP) to improve the flexibility of message delivery into Kafka. Customers can make use of a number of integrated predefined output formats, or adapt these user exits to define their own custom formats, what data is included in the message payload, and more. Apart from giving users the flexibility of defining the message formats and payloads, users are now also able to specify the Kafka topic names for their message destinations, and much more.

Moreover, as more common customer needs are assessed IBM will add more such predefined output formats.

For more information on the KCOP and samples available, please see the IIDR (CDC) knowledge center:

https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.4.0/com.ibm.cdcdoc.cdckafka.doc/concepts/kafkakcop.html

For demo videos on how to make use of the sample KCOPs or to contribute to the IBM Data replication community, please see the replication developer works page: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W8d78486eafb9_4a06_a482_7e7962f5ac59/page/Replication%20How%20To%20Videos

For more information on how IBM Data Replication can provide near real-time incremental delivery of transactional data to your Hadoop and Kafka based Data Lakes , visit : https://www.ibm.com/analytics/data-replication

Blog Post by : Davendra Paltoo, Offering Manager, Data Replication

Follow him on Twitter : https://twitter.com/Davendr18397388

One thing that is common across organizations today is that each one wants to be customer-centric and in order to be so, they need to be insights-driven. Insights-driven firms are growing at an average of more than 30% annually and are on track to earn $1.8 trillion by 2021 predicts Forrester.

These Insights-driven organizations are built on data. In order to have 360-degree view of a customer, organizations need data which is spread across disparate databases and data warehouses. Some data may reside on premise in a IBM DB2 database, or Teradata database or in the cloud in a Microsoft Azure SQL based on business needs.

Irrespective of where your data resides, you need a data replication solution for real time replication requirements in support of your data integration and analytics projects.

To help meet the needs of organizations who make use of Teradata and MS Azure SQL databases, IBM Data Replication Change Data Capture technology now supports targeting Teradata in the latest 11.4 product line. Previously, the CDC apply for Teradata was only available in earlier supported versions.

In addition, CDC is now validated to support targeting Azure SQL databases, which closely resemble Microsoft SQL Server databases, via use of the CDC for Microsoft SQL Server data replication target/apply.

CDC Replication supports Azure SQL Database as a remote target only. The CDC Replication target can either be installed on premises or in an Azure VM. For optimum performance, the CDC Replication target should be installed on a VM in the same region as the Azure SQL Database.

For more information, on our Azure SQL database targeting capability please see our knowledge centre:

https://www.ibm.com/support/knowledgecenter/SSTRGZ_11.4.0/com.ibm.cdcdoc.cdcformssql.doc/concepts/configuringazuresql.htm

For more information on our Teradata apply in 11.4 visit our knowledge centre:

https://www.ibm.com/support/knowledgecenter/SSTRGZ_11.4.0/com.ibm.cdcdoc.cdcforteradata.doc/concepts/whatsnew.html

Blog post by: Davendra Paltoo, Offering Manager, Data Replication

Follow him on twitter: https://twitter.com/Davendr18397388

In addition to Apache Kafka’s  more widely known capabilities as a distributed streaming platform, its capabilities, scalability and low cost as a storage system make it suitable as a central point in the enterprise architecture for data to be landed and then consumed by various applications.

When messages are landed in a Kafka cluster, there are a variety of available connectors or consumers that can in turn retrieve messages and deliver such messages to target destinations such as HDFS and Amazon S3.  Or, Kafka users can write their own consumers.

However, developers of consumers struggle to find an easy way to:

• Only retrieve and consume transactions that have been completely delivered to Kafka (possibly to many different Kafka topics) with records in the original order as they occurred on the source database.
• Avoid processing of duplicate messages that have been delivered to Kafka.
• Avoid deadlocks when reading committed transactions from Kafka topics.

This is a concern to many Kafka users because in some critical scenarios, it is extremely valuable to users to have Kafka behave with database-like transactional semantics.

For example:

• The Kafka consumer needs to use Kafka data to populate  parent child referential integrity tables in the same order as they were populated on the source.
• Processing of duplicate messages (which can sometimes occur during some failure and recovery scenarios when messages are being delivered to Kafka by Kafka producers or writers) cannot be tolerated by downstream applications. For example, knowing that no duplicates will be delivered, key business events can be triggered exactly once in response to messages delivered into Kafka.
• The Kafka consumer needs to guarantee retrieval and delivery of consistent transactions with the ability to recover from failures.

Why settle for duplicate data and promises of eventual consistency when you can leverage the performance and low cost of Kafka AND have database-like transactional semantics while not compromising on performance while delivering changes into Kafka?

IBM Data Replication’s “CDC” technology, with the initial version of its 11.4.0 release, provided users the ability to replicate from any supported CDC Replication source to a Kafka cluster by using the CDC target Replication Engine for Kafka.
In a recent delivery update, CDC now provides a java class library that can be included in a Kafka consumer application that is intended to consume data delivered by CDC into Kafka. This library, provides:

• Data in the original source log stream ORDER with identifiers available to denote transaction boundaries.
• A mechanism for ensuring exactly once delivery, so if there is an interruption in the Kafka environment and data has to be re-sent to Kafka by a producer or writer into Kafka, a consumer can be developed to only consume and process the data once.
• A “bookmark “that can be used to restart the consuming application from where it last left off processing.

Also available in the recent CDC delivery, are sample consuming applications that show how to:

• Poll records that were read by the Kafka transactionally consistent consumer for a specified subscription and write them to the standard output in the order of the source operation.
• Poll records that were read by the Kafka transactionally consistent consumer and publishes them in text format to a JMS topic.

Users are free to adapt the samples to suit their needs or to write their own consumer applications.
For more information on the Kafka transacationally consistent consumer please see our knowledge center at:
https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.4.0/com.ibm.cdcdoc.cdckafka.doc/concepts/kafkatcc.html

For demo videos on how to make use of the IBM Data Replication Kafka Apply or to contribute to the IBM Data replication community, please see the replication developer works page: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W8d78486eafb9_4a06_a482_7e7962f5ac59/page/Replication%20How%20To%20Videos

For more information on how IBM Data Replication can provide near real-time incremental delivery of transactional data to your Hadoop and Kafka based Data Lakes or Data Hubs, download this solution brief. 

Blog Post by : Davendra Paltoo, Offering Manager, IBM Data Replication

In November 2017, IDC’s Data Integration and Integrity (DII) Software Research Group conducted a survey targeted at end-users of DII software, including similar questions as asked in 2015 to help identify trends. With the need to bring real time and most recent data in to the enterprise for analytics efforts, IDC survey respondents found that keeping data synchronized among applications is and will be the most prevalent use case for data integration with 51% of survey respondents indicating application data sync to be the top use case. IDC is also observing that data intelligence will grow from one of the least today, to one of the most prevalent by 2020, in support of data governance, profiling, discovery and knowledge¹.

As organizations continue to face the challenge of bringing real time data for analytics applications/purposes, IBM provides the IBM® Integrated Analytics System (IIAS) which consists of a high-performance hardware platform and optimized database query engine software that work together to support various data analysis and business reporting features for today’s big data needs. IBM also provides Db2 Warehouse for Data Warehousing to provide users with in database analytics capabilities.

Users often need to integrate data from various data sources into their IIAS appliance or Db2 Warehouse deployment, if such technologies exist in their enterprise.

In addition, for an increasing variety of analytics use cases, only the freshest data is sufficient. Whether it is the customer interacting with a self-service portal or an executive looking for up to the minute financial performance, no organization can afford to serve up stale data. Yet, this can happen if organizations depend on periodic bulk movement of data around the enterprise. 

IBM Data Replication provides up to the second replicas of changing data where and when needed.  Our users are replicating operational data to everything from a traditional data warehouses to a data appliance such as the Pure Data Analytics (PDA) appliance or IIAS, to a Big Data cluster driven by Apache Kafka and Hadoop or even to a Cloud based OLAP environment such as Db2 Warehouse.

IBM data replication (Change Data Capture technology) can deliver changes using log based captures that minimize the impact on source databases from ALL supported CDC sources into IIAS and Db2 warehouse directly (i.e. in one hop).

In the recent release, IBM has introduced a new Mirror Bulk Apply option that supports Db2 External Tables as the apply mechanism for faster ingest into column organized tables within Db2 Warehouse deployed in the IIAS appliance or column organized tables in “standalone”  Db2 Warehouse databases. This is as compared to the previously available apply mechanisms for applying changes to such column organized tables. External table bulk apply is the algorithm that CDC employs to apply changes to the IBM Pure Data Appliance or “Netezza”.

This support is now being extended to Db2 Warehouse. Such column organized tables are useful in databases intended for use in analytics since they aid query performance.

The new CDC apply performance capability will give end users the confidence that even the data from the most high volume transactional systems can be replicated with acceptable latency into IIAS and Db2 Warehouse’s column organized tables.

For more information on the Mirror Bulk apply capability please see our knowledge center.

For more information on IBM Data replication, please read the IBM Data Replication solution brief.

[1] IDC Data Integration and Integrity Survey, Nov 2017

Blog By: Davendra Paltoo, Offering Manager, IBM Data Replication

Every business today revolves around data. Conversations with our customers frequently confirm the following:

1. Organizations need real time access to the latest data to improve organizations’ customer experiences and also to make timely business decisions. Continuous Intelligence requires nothing but the real-time availability of data. Gartner predicts that by 2022, more than half of major new business systems will incorporate continuous intelligence that uses real-time context data to improve decisions. Moreover, Gartner also predicts that augmented analytics that uses machine learning and AI techniques will transform how analytics content is developed, consumed and shared.
2. Our users continue to experience challenges with making data easier to access and use especially when organizations have multiple databases for multiple purposes and often when organizations have some data on premise and some in the cloud. Moreover, with open source database management systems “OSDBMS” growing in popularity due to lower expected costs and improving quality¹ , Gartner forecasts that “By 2022, more than 70% of new in-house applications will be developed on an OSDBMS, and 50% of existing commercial RDBMS instances will have been converted or will be in process of converting.”²

Organizations are looking cost effective solutions to the above challenges.

IBM Data Replication “IDR” can help by providing up to the second replicas of changing data where and when needed keeping data synchronized with low latency. Our users are replicating operational data from most of the world’s popular relational databases like Oracle and Db2 z/OS to everything from a traditional data warehouses to a data appliance such as Pure Data Analytics or IBM Integrated Analytics System, to a Big Data cluster driven by Apache Kafka and Hadoop, or even to a Cloud based On Line Analytical Processing (OLAP) environment such Db2 Warehouse on Cloud.

Understanding the adoption patterns around OSDBMS, for some time, IBM Data Replication has provided users the ability to feed data INTO PostgreSQL while sourcing from a wide variety of source DBMS by employing low impact database log based captures.

With Db Engines currently indicating that PostgreSQL is currently the fourth most popular database in the world, leveraging PostgreSQL around the enterprise just got easier. IBM Data Replication now supports PostgreSQL as a SOURCE³ in a recent deliverable.

The IDR CDC PostgreSQL capture interoperates, out of the box, with the extensive array of target platforms supported by CDC. This includes most major DBMSs, Kafka, Hadoop, files, messaging systems and more.

PostgreSQL as a source released in an IDR 11.4 Fixpack with support for both PostgreSQL Enterprise and community editions provided they meet the published system requirements . Note that users with sufficient license entitlement of either IBM Data Replication or IBM InfoSphere Data Replication can deploy the new PostgreSQL capture with no additional purchase. If in doubt, please check with your IBM account representative.

Click here for a table of contents linking to more details about IBM’s CDC technology for capturing PostgreSQL database changes and replicating them across the data center or around the world.
 

¹ https://opensource.com/life/15/12/why-open-source

² https://www.exist.com/blog/the-future-is-open-edb-postgres-and-your-enterprise-data/)

³ https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.4.0/com.ibm.cdcdoc.postgresql.doc/concepts/overview.html

⁴ https://www.ibm.com/developerworks/community/blogs/a9b542e4-7c66-4cf3-8f7b-8a37a4fdef0c/entry/IIDR_11_4_0_2_5098_for_all_LUW_engines1?lang=en