IT02997: Sensitive DB Updates to standby server CAUSES HA RELATIONSHIP TO DISCONNECT

Tivoli Storage Productivity Center Refresh Pack 4
Fix Pack 5.2.4 (November 2014) for Tivoli Storage Productivity Center
Fix Pack 5.2.4.1 (December 2014) for Tivoli Storage Productivity Center
Refresh Pack 5.2.5 (March 2015) for Tivoli Storage Productivity Center (withdrawn)
Fix Pack 5.2.5.1 (April 2015) for Tivoli Storage Productivity Center (withdrawn)
Refresh Pack 5.2.6 (June 2015) for Tivoli Storage Productivity Center
Refresh Pack 5.2.3 (August 2014) for Tivoli Storage Productivity Center
Refresh Pack 5.2.7 (August 2015) for Tivoli Storage Productivity Center
IBM Spectrum Control V5.2.8 (December 2015)
IBM Spectrum Control V5.2.9 (February 2016)
IBM Spectrum Control V5.2.10 (May 2016)
IBM Spectrum Control V5.2.10.1 (July 2016)
IBM Spectrum Control V5.2.11 (August 2016)
IBM Spectrum Control V5.2.12 (November 2016)
IBM Spectrum Control V5.2.13 (March 2017)
IBM Spectrum Control V5.2.14 (May 2017)
IBM Spectrum Control V5.2.15 (August 2017)
IBM Spectrum Control V5.2.15.2 (November 2017)
IBM Spectrum Control V5.2.16 (March 2018)
IBM Spectrum Control V5.2.17 (May 2018)
Fix Pack 5.2.7.1 (February 2016) for Tivoli Storage Productivity Center

APAR status

• Closed as fixed if next.

Error description

• The GUI panel passes in an invalid statement when the panel
  is left open during a refresh.
  This causes the HA communication to receive a SQL exception
  causing the relationship to think it has gone disconnected.
  The relationship will go back to synchronized on the next
  valid update to the HA standby embedded datasore.
  
  
  Example:
  
  01-Jul-2014 14:25:45 : syj0 : IWNR3091E : High-Availability had
  a connection failure with the server xxxx.xxxx.xxxx.com with a
  message code of 129 and a reason code of 0.
  
  ================================================================
  
  
  2014-07-01 14:25:45.463+0000 Thread-40214 RepMgr D
  DBCmdHandler handleSQLException DEBUG: SQLState = XCL12
  2014-07-01 14:25:45.463+0000 Thread-40214 RepMgr D
  DBCmdHandler handleSQLException DEBUG: Unsupported SQL State:
  XCL12
  2014-07-01 14:25:45.463+0000 Thread-40214 RepMgr E e
  DBCmdHandler handleSQLException java.sql.SQLException: An
  attempt was made to put a data value of type 'java.lang.String'
  into a data value of type 'BLOB'.
   at
  org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException
  (Unknown Source)
  

Local fix

• A reconnect command will force the High Availability
  relationship to return to synchronized.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Users running TPC-R 5.1.1.5 or above who have a high         *
  * availability relationship to another TPC-R server and do
  * one of the following:
  * - change to the active server credentials for a device
  * - update site location
  * - restart TPC-R active server
  *
  * After any of these actions are taken, HA standby will
  * show disconnected from the storage due to an authentication
  * error.
  *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * | fix pack | 5.2.4-TIV-TPC-FP0000 - target 3Q 2014 |         *
  *                                                              *
  *                                                              *
  * http://www-01.ibm.com/support/docview.wss?&uid=swg21320822   *
  *                                                              *
  * The target dates for future fix packs do not represent a     *
  * formal                                                       *
  * commitment by IBM. The dates are subject to change without   *
  * notice.                                                      *
  *                                                              *
  * This issue was introduced in version 5.1.1.5 and causes the  *
  * active server to update the standby server with invalid      *
  * sensitive information (credentials) and the standby will no  *
  * longer be able to connect to the storage connection after    *
  * any change has been made to credential information on the    *
  * active server.                                               *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Update to the aforementioned TPC-R release to obtain a  fix. *
  * To mitigate the risk, disconnect the TPC-R HA relationship
  * via remove standby from the active server prior
  * to engaging in any of the activities listed in
  * Users Affected section.
  * After action is completed HA relationship can be
  * re-established
  ****************************************************************
  

Problem conclusion

• The data on the standby will now be updated properly and HA
  standby server will be able to connect to storage after a change
  has been made.
  

Temporary fix

• First ensure that the Active server has stable connections to
  the DS8000 devices. If it does not - this may be a different
  issue - if thats the case contact support.
  
  The workaround for this apar is to follow this
  procedure to clear out the standby database:
  1) On the STANDBY server, issue a takeover
  2) Stop the TPC-R server (see infocenter for instructions for
  your particular installation and precautions regarding Heartbeat
  and MM)
  3) Delete this directory where the TPC-R database
  for 5.1.1.5
  {install_dir}/ewas/profiles/ReplicationServerProfile/database/
  csmdb
  
  for 5.2
  {install_dir}/wlp/usr/servers/replicationServer/database/csmdb
  4) Start the TPC-R server (see infocenter for instructions for
  your particular installation)
  5) Re-establish the HA relationship
  6) The active and standby will synchronize with correct
  credentials for the DS8000 devices
  
  * Note this is only a temporary solution. Any of the following
  actions on the active server will cause the devices on the
  standby to again go to a disconnected state
   - update to device on the active server (only disconnects the
     device updated on active)
   - change of site location (only disconnects the device updated
     on active)
   - restart of TPC-R (all devices on standby go disconnected)
  

Comments

• If the user has hit this error HA capabilities are still
  possible. If there is a site
  outage on the active server the user can still issue takeover
  command from the standby. A takeover to the
  standby TPC-R server will now require a few extra steps
  to be
  taken After Takeover to a standby that is experiencing IT02297.
  
  Here are the additional steps after takeover to the standby:
  1) For each DS8k showing disconnected, enter the correct
  password into the connection details panel.
  2) Click OK, the connection should be successful and show
  connected
  3) The user can now issue commands to the storage as normal
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PI21609

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

• R510 PSY

     UP

• R520 PSY

     UP