Using multiple file transfer queues when replicating column-organized tables

You might improve replication performance for column-organized tables when the network is a bottleneck by using multiple file transfer queues for data transmission. This configuration might be particularly beneficial when the replication source and target are hundreds or even thousands of miles apart.

Before you begin

The user ID that performs this task must have dsadm authority on the source database.

About this task

When you invoke this feature, replication automatically creates up to four pipelines, each with its own IBM® MQ send queue, receive queue, and channel, to fully saturate the bandwidth between the source and target systems. It enables highly parallel file transfer that is especially effective when the systems are separated by long distances.

Using a single pipeline requires the source capture process to wait for the target apply process to acknowledge receipt of a file that contains the replicated data. With multiple pipelines, the capture process can continue to send files by writing to the send queues in random order without needing to wait for acknowledgment. This boosts replication performance when handling massive table inserts of hundreds of thousands of rows at a time.

The default parallel degree is 4. The parallel degree feature uses a script, bludr-config-mq-file-transfer-degree.sh, that enables you to set or change the degree or display the current degree that is being used. For the full syntax, see bludr-config-mq-file-transfer-degree.sh script. You can also set or change the degree of parallelism by using the replication REST APIs.

Procedure

  1. Use a Docker container bash shell to invoke the bludr-config-mq-file-transfer-degree.sh script.

    For example, the following command invokes the script to set a parallel degree of 3 for the replication set RS1:

    /opt/ibm/bludr/scripts/bin/bludr-config-mq-file-transfer-degree.sh -u bluadmin -c RS1 -d 3

    After you enter the password, the replication set is stopped after ensuring that all data at the target was applied. The script then creates the following MQ objects:

    Source system
    File transmission queue
    File send queue
    File sender channel
    Target system
    File receiver channel
    File receive queue

    The replication set is then returned to active state and replication continues. If the set was stopped, it is left in inactive state.

    A log, /mnt/blumeta0/bludr/logs/server/bludr-config-mq-file-transfer-degree_timestamp.log, is generated to help you troubleshoot any issues with the script. The timestamp value is in the format yymmddhhmmsss.

  2. Optional: To set a different parallel degree, invoke the script again with a different setting for the -d parameter.
  3. Optional: To display the current parallel degree, invoke the script with the -l parameter.
    For example:
    /opt/ibm/bludr/scripts/bin/bludr-config-mq-file-transfer-degree.sh -u bluadmin -l RS1