What's new in Version 11.4

The following new features were added to the CDC Replication engines and other components in IBM® Data Replication Version 11.4 builds.

You can find the latest builds on IBM Fix Central, and see the latest full list of features and defect fixes for CDC Replication engines and Management Console and Access Server on the IBM support site.

Management Console and Access Server

  • Access Server now supports TLS encryption between clients and datastores. In addition to negotiating encryption, Access Server supports connecting to datastores that always have TLS encryption without negotiation. You can enable this feature by setting datastoresAlwaysTLS=true in the tls.properties file. When this feature is enabled, Access Server cannot communicate with datastores that negotiate encryption through STARTTLS.
  • Management Console and Access Server support Windows 10. Access Server supports Linux® for System z®.
  • CDC Replication no longer requires a centralized Access Server and provides the option to use Management Console or CHCCLP as standalone products, each with an Embedded Access Server.
  • Access Server and Management Console support Lightweight Directory Access Protocol (LDAP). Traditionally, Access Server authenticates users, stores user credentials and access data information, and acts as the centralized communicator between all replication agents and Management Console clients. In an LDAP configuration, the LDAP server manages the user credentials and authenticates users, and it can manage access data information depending on the LDAP configuration that you choose. Traditional functionality remains available.

All Linux, UNIX, and Windows engines

  • You can now use TLS to encrypt communication between the source and target in all CDC Replication Java™ engines. You can use self-signed certificates, a central self-signed certificate authority, or a public certificate authority to create private key stores and trust stores within CDC Replication, or use third-party tools such as openssl.

    To create and manage encryption profiles that are needed to enable or disable encryption, you can use Management Console or the command-line configuration utility. CDC Replication supports requiring TLS encryption for all connections from Access Server and other datastores. CDC Replication supports always encrypting communication with TLS, with or without negotiation.

    You can enable encryption without negotiation by setting Engine-to-Engine Communication Encryption to Always in the encryption profile. When this feature is enabled, CDC Replication cannot communicate with source datastores, target datastores, or Access Servers that negotiate encryption through STARTTLS.

CDC Replication Engine for Db2 for z/OS Remote Source

  • The CDC Replication Engine for Db2® for z/OS® Remote Source is a new replication engine as of version 11.4.0.1. This engine provides full source-side replication capability for Db2 for z/OS with a minimal footprint on z/OS. The replication engine is installed on a distributed platform (Linux, UNIX, or Windows) and after initial setup is entirely administered and maintained on that platform.

    The product’s interactions with Db2 for z/OS are through standard JDBC interfaces using a product-supplied stored procedure to obtain Db2 for z/OS log data. After a one-time setup of the stored procedure environment, no operator or administrator involvement is required directly on z/OS. Upgrades to the product, including applying maintenance to the stored procedure code, are performed entirely on the distributed platform without the need for Db2 or z/OS-specific skills. Any needed updates to the stored procedure are performed automatically and securely by the engine through SSH.

    Samples are provided for granting the required z/OS and Db2 privileges and testing that the privileges were granted. The samples are found in the samples directory in your CDC Replication Engine for Db2 for z/OS Remote Source installation directory.

    You can manually install the external stored procedure if you prefer. If the external stored procedure for the current product version was installed correctly, it is not reinstalled.

    You can configure an instance with one user ID that has the privileges to install the external stored procedure and then reconfiguring the instance with a different user ID without the privileges to install the stored procedure. This method allows CDC Replication to run with the minimum privileges that are needed for replication, without the privileges that are needed for installation.

Data Replication for VSAM for z/OS Remote Source

  • IBM Data Replication VSAM for z/OS Remote Source is a CDC Replication engine that enables replication of VSAM source data with a minimal footprint on z/OS. The replication engine is distributed as a Linux container, which you use to set up, maintain, and administer the replication environment. Learn more ...

CDC Replication Engine for Db2 Database

  • CDC Replication supports external table mirror bulk apply, a new way to apply data to IBM Integrated Analytics System (IAS) databases that uses the external table feature that is available for IAS. You can configure this feature through user exits in a fashion similar to the CDC Replication Fast Apply feature.
  • You can replicate from Db2 Warehouse and Db2 Warehouse on Cloud database row-organized tables. Replicating from column-organized tables is not yet supported.
  • You can now use a JDBC Type 4 connection between the replication engine and Db2. Among the benefits:
    • For a remote configuration, you no longer need to catalog the database if you use JDBC refresh. For Fastload, you still must catalog the database.
    • When a database is SSL-enabled, you can configure JDBC Type 4 advanced parameters to secure all CDC Replication connections to the database.
    • Using the Type 4 connection, you can leverage Db2 features such as Work Load Balancing that is used by PureScale databases.

    For more information, see Adding an instance of the CDC Replication Engine for Db2 Database.

CDC Replication Engine for FlexRep

  • You can now replicate to EnterpriseDB version 10 and 11 targets.
  • You can replicate changes to MySQL 5.7 and 8, to PostgreSQL 9.6, 10, and 11, and to Amazon Redshift. The replication engine distributes the Redshift JDBC driver from Datadirect. See Supported JDBC drivers for more details.

CDC Replication Engine for Kafka

  • The CDC Replication Engine for Kafka is now available on Linux for IBM System z.
  • The Kafka transactionally consistent consumer is available. By including the Kafka transactionally consistent consumer library in your Java applications, you can:
    • Recreate the order of operations in source transactions across multiple Kafka topics and partitions.
    • Consume Kafka records that are free of duplicates

    For more information, see Kafka transactionally consistent consumer.

  • The engine supports Kafka custom operation processors (KCOP) that can enable you to perform the following functions:
    • Write audit records in comma-separated values (CSV)
    • Write records in JSON. No schema registry is required.
    • Specify topic names and set up mappings between multiple source tables and target topics.
    • Write audit records in Avro format and register the schema in a Confluent schema registry or a Hortonworks Schema Registry Service.
    • Write an audit of before and after images . You can specify content for the header field that is optional in a Kafka producer record for Apache Kafka 0.11 and higher.

CDC Replication Engine for Microsoft SQL Server

  • You can use the CDC Replication Engine for Microsoft SQL Server to replicate to a Azure SQL Managed Instance or to Microsoft Azure SQL Database target. CDC Replication supports Azure SQL Database as a remote target only. The CDC Replication target can either be installed on premises or in an Azure VM, ideally on a VM in the same region as the Azure SQL Database. For more details, see Configuring the CDC Engine for Microsoft SQL Server for Azure SQL Database. You can use the CDC Replication Engine for Microsoft SQL Server to replicate from a Microsoft SQL Server database that is deployed in a Microsoft Azure VM. The engine supports Microsoft SQL Server 2017 as a source or a target.
  • You can configure a CDC Replication Engine for Microsoft SQL Server that is remote from the source database, where the engine does not have direct access to the database transaction logs. For more information, see Configuring a remote source of the CDC Replication Engine for Microsoft SQL Server.
  • The engine uses dynamic online virtual log file discovery based on user authorization. Virtual log file discovery switches to DBCC LOGINFO when the CDC Replication user has SYSADMIN privileges.
  • You can configure CDC Replication Engine for Microsoft SQL Server with Capture table-based configuration mode using a database that supports Microsoft SQL CDC from:
    • Transparent data encryption(TDE) enabled database
    • Databases with log compression enabled
    • Azure Managed SQL Databases
    • Remote databases

    To use this configuration mode, the CDC Replication instance user must be permitted database ownership.

CDC Replication Engine for MySQL sources

  • The CDC Replication Engine for MySQL sources is introduced to provide full source-side replication capability for MySQL. CDC Replication reads MySQL binary logs in a standard way and interacts with MySQL through JDBC. MySQL Community Edition is now supported as a replication source.

CDC Replication Engine for Oracle databases

  • The engine supports Oracle Autonomous Data Warehouse and Oracle OLTP as targets.
  • The engine supports Oracle 12c Release 2, Oracle 18c, and Oracle 19c.
  • Replicating Oracle SecureFile LOB data is now supported, as is replicating from source tables that use the Oracle interval partition feature, which automatically adds partitions. This engine also supports interval partition functionality that includes range partitions in Oracle versions such as 12c Release 2.
  • The engine supports the Oracle system parameter exafusion_enabled, which is used on Linux Exadata systems that run Automatic Storage Management (ASM).
  • The CDC Replication Engine for Oracle databases can replicate data that uses Transparent Data Encryption. Both column encryption and table space encryption are supported. Transparent Data Encryption is supported from Oracle 11gR2 and later.
  • You can install the CDC Replication Engine for Oracle databases on any supported platform (Linux/AIX®/Windows) and use a remote log reading configuration to process logs from an Oracle database that is installed on a different operating system (Solaris/HPUX/AIX/Linux/Windows). The operating systems for the CDC Replication Engine for Oracle databases and the database no longer need to have the same endianness (they no longer need to both be little endian or both big endian). The new mirror_redo_log_format system parameter enables you to indicate the operating system on which the Oracle redo log is generated.

CDC Replication Engine for PostgreSQL sources

  • The new CDC Replication Engine for PostgreSQL sources provides full source-side replication capability for PostgreSQL databases. The replication engine is installed on a distributed platform (Linux, UNIX, or Windows) and after initial setup is entirely administered and maintained on that platform. The product interacts with PostgreSQL through a standard JDBC interface by using a product-supplied replication API to obtain PostgreSQL log data.
  • PostgreSQL running on Amazon Relational Database Service (RDS) is now supported as a replication source. Replication of PostgreSQL 11 is now supported.

CDC Replication Engine for Teradata

  • The CDC Replication Engine for Teradata now supports Teradata 16.00, 16.10, and 16.2.