IT04424: HA SYNCHRONIZATION QUEUE CAN HIT THE LIMIT WITH LARGE AMOUNTS OF COPYSETS.

Tivoli Storage Productivity Center Refresh Pack 4
Fix Pack 5.2.4 (November 2014) for Tivoli Storage Productivity Center
Fix Pack 5.2.4.1 (December 2014) for Tivoli Storage Productivity Center
Refresh Pack 5.2.5 (March 2015) for Tivoli Storage Productivity Center
Fix Pack 5.2.5.1 (April 2015) for Tivoli Storage Productivity Center (withdrawn)
Refresh Pack 5.2.6 (June 2015) for Tivoli Storage Productivity Center
Refresh Pack 5.2.7 (August 2015) for Tivoli Storage Productivity Center
IBM Spectrum Control V5.2.8 (December 2015)
IBM Spectrum Control V5.2.9 (February 2016)
IBM Spectrum Control V5.2.10 (May 2016)
IBM Spectrum Control V5.2.10.1 (July 2016)
IBM Spectrum Control V5.2.11 (August 2016)
IBM Spectrum Control V5.2.12 (November 2016)
IBM Spectrum Control V5.2.13 (March 2017)
IBM Spectrum Control V5.2.14 (May 2017)
IBM Spectrum Control V5.2.15 (August 2017)
IBM Spectrum Control V5.2.15.2 (November 2017)
IBM Spectrum Control V5.2.16 (March 2018)
IBM Spectrum Control V5.2.17 (May 2018)
Fix Pack 5.2.7.1 (February 2016) for Tivoli Storage Productivity Center

APAR status

• Closed as fixed if next.

Error description

• When the HA synchronization queue hits the limit of 16384
  objects, the HA relationship will disconnect with the following
  message:
  IWNR3091E [2014-09-07 18:02:04.752-0400]
  High-Availability had a connection failure with the server
  server_name.com with a message code of 122 and a reason code of
  16.
  IWNR3090E [2014-09-07 18:02:04.768-0400]
  High-Availability active message queue was full so is no longer
  able to send updates to the standby at server_name.com.
  
  And the following exception is seen on the logs:
  2014-09-07 18:01:59.619-0400 AsyncDatabaseSynchronizer
  RepMgr E e com.ibm.csm.server.ha.HaActiveConnMgr
  sendQueuedUpdateCmd(DbSyncMsg)
  com.ibm.csm.server.ha.HaQueueFullException:
  The HA QUEUE is FULL with 16384 objects
  
  This apar will allow the limit to be tuned in the
  rmserver.properties to ensure customers with large amounts of
  copysets can maintain HA server capability.
  

Local fix

• The HA relationship must be removed and re-established in order
  for the active and standby be synchronized state.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All users who have large amounts of pairs being managed by   *
  * TPC-R and have a High Availability standby server            *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * | fix pack | 5.2.4-TIV-TPC-FP0000 - target 4Q 2014 |         *
  *                                                              *
  *                                                              *
  * http://www-01.ibm.com/support/docview.wss?&uid=swg21320822   *
  *                                                              *
  * The target dates for future fix packs do not represent a     *
  * formal                                                       *
  * commitment by IBM. The dates are subject to change without   *
  * notice.                                                      *
  *                                                              *
  * The limit was hard coded in very early releases of TPC-R.    *
  * This number has since become inadequate to handle the large  *
  * scale environments in use today. This apar will increase the *
  * queue size by a factor of 2^3 and allow the limit to be set  *
  * via a property file if larger configurations are needed.     *
  * Special consideration has to be taken when increasing the    *
  * value too high as more memory will be needed the greater     *
  * this queue size is. The queue size does not grow or shrink   *
  * and is static.                                               *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * none                                                         *
  ****************************************************************
  

Problem conclusion

Temporary fix

• If the following error is hit:
  IWNR3089E : High-Availability had a synchronization failure with
  the server xxxx when trying to become highly available with a
  message code of 114 and a reason code of 0.
  
  Attempt to re-establish the HA relationship. If the procedure
  does not work contact support for instructions to increase the
  limit or install 5.2.4.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

• R520 PSY

     UP