IT11056: CRASHED TIMER CAUSES ERROR IWNR2999E FOR SEVERAL TPC-R OPERATIONS

IBM Spectrum Control V5.2.8 (December 2015)
IBM Spectrum Control V5.2.9 (February 2016)
IBM Spectrum Control V5.2.10 (May 2016)
IBM Spectrum Control V5.2.10.1 (July 2016)
IBM Spectrum Control V5.2.11 (August 2016)
IBM Spectrum Control V5.2.12 (November 2016)
IBM Spectrum Control V5.2.13 (March 2017)
IBM Spectrum Control V5.2.14 (May 2017)
IBM Spectrum Control V5.2.15 (August 2017)
IBM Spectrum Control V5.2.15.2 (November 2017)
IBM Spectrum Control V5.2.16 (March 2018)
IBM Spectrum Control V5.2.17 (May 2018)

APAR status

• Closed as program error.

Error description

• An internal timer thread utilized by TPC-R code crashes without
  detection. When invoking certain operations (such as "Refresh
  Config"),
  the following errors appear in the csmTrace logs:
  ''''''''''''''''
  2015-08-18 16:06:49.044+1000 CSMCOM-E28B RepMgr D
  com.ibm.csm.server.hw.ess.ccw.ESSElementCatalogPopulator
  refresh() TRACE: (DS8000:BOX:xxxx.xxxxx) Element Catalog
  Populator Refresh has been issued.; isWorking=false
  2015-08-18 16:06:49.044+1000 CSMCOM-E28B RepMgr D
  com.ibm.csm.util.thread.SchedulerService cancel TRACE: looking
  for
  com.ibm.csm.server.hw.ess.ccw.ESSElementCatalogPopulator@7b83b93
  e [DS8000:BOX:xxxx.xxxxx] and found
  task=com.ibm.csm.server.hw.ess.ccw.ESSElementCatalogPopulator@7b
  83b93e [DS8000:BOX:xxxx.xxxxx]
  2015-08-18 16:06:49.044+1000 CSMCOM-E28B RepMgr E e
  com.ibm.csm.server.communication.ClientRequest
  processCommandParcel java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
  Method)
      [...]
      at java.lang.Thread.run(Thread.java:804)
  Caused by: java.lang.IllegalStateException: Timer already
  cancelled.
      at java.util.Timer.sched(Timer.java:408)
      at java.util.Timer.schedule(Timer.java:219)
      [...]
  2015-08-18 16:06:49.044+1000 CSMCOM-E28B RepMgr I
  com.ibm.csm.server.communication.ClientRequest
  processCommandParcel IWNR2999E [18/08/2015 4:06:49 PM] A Tivoli
  Storage Productivity Center for Replication internal error
  occurred.
  ''''''''''''''''
  In this condition, TPC-R results in error IWNR2999E "internal
  error occurred" when performing several operations such as:
  - Refresh Config
  - HA disconnect/reconnects
  

Local fix

• restart TPC-R service.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Users of the product who experience a crash of the Timer     *
  * thread which is used by all pollers, loggers, work tasks,    *
  * and other functions that require schedule.                   *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * | fix pack | 5.2.8-TIV-TPC-FP0000 - target 4Q 2015 |         *
  *                                                              *
  *                                                              *
  * http://www-01.ibm.com/support/docview.wss?&uid=swg21320822   *
  *                                                              *
  * The target dates for future fix packs do not represent a     *
  * formal                                                       *
  * commitment by IBM. The dates are subject to change without   *
  * notice.                                                      *
  *                                                              *
  * This apar addresses an unexpected crash of the Timer thread  *
  * that is used by many functions in the TPC for Replication    *
  * product. The thread crash will cause pollers, loggers, work  *
  * tasks, and other functions that require schedule not to be   *
  * able to be scheduled. Some of the problems that can occur if *
  * the thread is not running will not be seen directly but may  *
  * manifest in problems with running commands, session may get  *
  * stuck in preparing, or the storage  / copyservice changes    *
  * caused by External failover may not work properly. The       *
  * particular problem that this surfaced with was an internal   *
  * error while running the refresh config. Although outOfMemory *
  * issues are very rare, this maintenance should be applied as  *
  * soon as possible to avoid future problems. Customers should  *
  * ensure that the TPC server has the recommended memory        *
  * http://www-01.ibm.com/support/docview.wss?uid=swg27039550    *
  * and amount of attached storage and copysets meets the heap   *
  * memory recommendation described in the infocenter            *
  * http://www-01.ibm.com/support/knowledgecenter/SSU9FR_5.2.7/c *
  * om.ibm.tpcrz.doc_5.2.7/frc_r_verifying_the_java_heap_size.ht *
  * ml?lang=en                                                   *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Ensure that the server meets the requirements listed         *
  * previously.                                                  *
  ****************************************************************
  

Problem conclusion

• The fix will detect the condition in which the timer thread has
  crashed and restart it.
  

Temporary fix

• Restart the TPC-R server.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TPC

• Fixed component ID

  5608TPC00

Applicable component levels

• R520 PSY

     UP