PH34713: LARGE UPDATE TRANSACTIONS WITH COMPENSATION (E.G. ROLLBACK TO SYNCPOINT) MAY LEAD TO HIGH REPLICATION LATENCY

APAR status

• Closed as documentation error.

Error description

• If transactions with a large number of updates on Db2 for z/OS
  (2+ million rows) were run including compensations such as
  rollback to syncpoint affecting 200,000+ log records, the
  replication latency of Integrated Synchronization may increase
  significantly (even to more than 45 minutes).
  
  Due to the updates and compensations on Db2for z/OS
  Integrated Synchronization will be busy with parsing the Db2
  log records without a need to apply anything on the target side.
  The latency will increase continuously until the log records
  belonging to the large transaction have been parsed completely.
  Afterwards the latency will decrease quickly to a normal level
  as shown by the corresponding SMF counters.
  
  Conclusion:
  Large update transactions in conjunction with compensating
  actions (e.g. rollbacks to syncpoint are the only type of
  transactions being handled slowly. The bottleneck goes back to
  the parsing of the compensation log records.
  Integrated Synchronization needs an improvement to avoid the
  high increase of replication latency.
  
  Additional keywords:
  InSync TS004943277 TS006781848 TS012530434
  DSNX881I W2021
  Github.../Everest/Customer-Cases/issues/135
  Github.../Everest/Customer-Cases/issues/259
  Github.../Everest/Customer-Cases/issues/498
  

Local fix

• Remove the tables involved in the transaction performing a
  large rollback from replication and afterwards restart the
  incremental update processes using the Accelerator console.
  Once the latency is back to normal, the tables can be added to
  replication again.
  

Problem summary

• Problem Summary:
  If transactions with a large number of updates on Db2 for z/OS
  (2+ million rows) are involving compensations (such as rollback
  to syncpoint) affecting 200,000+ log records, the replication
  latency of Integrated Synchronization may increase significantly
  (even to more than 45 minutes).
  
  The replication process retrieves update information from the
  Db2 for z/OS log and keeps it in memory until the corresponding
  changes are to be applied in the accelerator's database in
  Db2wh.
  The overall memory of an instance of Integrated Synchronization
  is limited (6 GB). The more "in flight" information for a
  transaction is kept in memory, the less memory remains
  available for the processing of other transactions. Less memory
  results in an increase of the replication latency.
  
  Users Affected:
  Users of replication based on Integrated Synchronization facing
  an increasing replication latency while a long-running
  transaction updating millions of Db2 for z/OS table rows is busy
  with a rollback.
  
  Problem Symptoms:
  See the APAR's Error description and Problem Summary above
  
  Problem Symptoms:
  The latency of the replication is increasing continuously from
  seconds to  even more than 45 minutes. Customer may see
  continuous occurrence of message DSNX881I W2021.
  

Problem conclusion

• Large compensations within transactions updating more than 2
  millions of rows in Db2 for z/OS impact the replication latency.
  To reduce the impact, the parsing of the log reader data needs
  to be improved as well as the consumption of memory.
  
  With Accelerator maintenance level 7.5.10 Integrated
  Synchronization has introduced
  - new data structures holding processing information (meta
    data) that result in a much faster processing of transaction
    related update information from Db2 for z/OS
  - a new customization parameter:
    UpdateCompensationMetadataToDiskMigrationThreshold.
  
    The threshold level specified defines a maximum number of
    update operations (insert/update/delete) for which
    processing data (meta data) are kept in memory. For update
    operations beyond the threshold the processing information
    is written to disk.
  
  Keep in mind:
  - A low threshold means less use of memory but more write to
    disk operations which impact the speed of replication.
  - A high threshold means increased consumption of memory causing
    replication latency to grow (significantly).
  - A consumption of the entire memory available will severely
    impact processing.
  
  As each Db2 transaction replicated consumes a base amount of
  memory, the calculation of the recommended value for
  UpdateCompensationMetadataToDiskMigrationThreshold is based on
  monitoring and evaluating the workload of replication.
  
  In case of facing message DSNX881I W2021, contact IDAA support
  by opening of a support case. The support engineers will
  evaluate the replication details and will set
  UpdateCompensationMetadataToDiskMigrationThreshold if
  recommended.
  
  We recommend that users of Integrated Synchronization upgrade
  the Accelerator to maintenance level 7.5.10.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels