IT39124: ANR9999D_0941050150 ALL CONNECTION POOLS ARE FULL SITUATION IS TRIGGERED BY PRIOR ANR0102E REPLICATION FAILURE

IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads

APAR status

• Closed as program error.

Error description

• ANR9999D_0941050150 InitConnectionPool(dbiconn.c:1709)
  Thread<532478>: All connection pools are full". is reported in
  the activity log indicating that all database connections are
  full.
  
  A key symptom of this APAR is replicate node failing with error
  below.  It is this replication failure which causes the
  incorrect management of the database connections.
  
  
  
  09/17/21   13:11:56      ANR0102E tbcli.c(2600): Error 1202
  inserting row in table
  
                           "Replicated.Objects". (SESSION: 199617,
  PROCESS: 416)
  
  09/17/21   13:11:56      ANR2183W nrfs.c(5042): Transaction
  0:234466260 was
  
                           aborted. (SESSION: 199617, PROCESS:
  416)
  
  
  Unmonitored, the database connections will steadily increase
  until error ANR9999D_0941050150 is reported to the activity log.
  
  
  
  08/18/2021 07:01:11      ANR9999D_0645605689
  tbOpenX(tbtbl.c:5499) Thread<532476>:
  
                           Failure participating on transaction.
  (SESSION: 286596,
  
                           PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR9999D Thread<532476> issued message
  9999 from:
  
                           (SESSION: 286596, PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x0000000100037c00 StdPutText
  
                           (SESSION: 286596, PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x0000000100038e6c OutDiagToCons
  
                            (SESSION: 286596, PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x000000010000d028 outDiagfExt
  
                           (SESSION: 286596, PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x000000010011fc64 tbOpenX
  
                           (SESSION: 286596, PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x0000000100bdc940
  
                           NrInsertHistoryRecord  (SESSION:
  286596, PROCESS: 672)
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x0000000100fa6224
  
                           NrReplicateFilespace  (SESSION: 286596,
  PROCESS: 672)
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x00000001005e5d7c
  
                           PcConsumerThread  (SESSION: 286596,
  PROCESS: 672)
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x00000001007ecf14
  
                           NrReplicateFilespaceStackTrackEntry
  (SESSION: 286596,
  
                           PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR9999D Thread<532476>
  0x00000001000114b0 StartThread
  
                           (SESSION: 286596, PROCESS: 672)
  
  
  08/18/2021 07:01:11      ANR0101E nrutil.c(452): Error 9991
  opening table
  
                           "Replication.History". (SESSION:
  286596, PROCESS: 672)
  
  08/18/2021 07:01:11      ANR9999D_0941050150
  InitConnectionPool(dbiconn.c:1709)
  
                           Thread<532478>: All connection pools
  are full. (SESSION:
  
                           286596, PROCESS: 672)
  
  
  IBM Spectrum Protect has a limit of 9600 concurrent database
  connections at any given time. When this APAR is seen,
  the following command can be used to monitor session use.
  
  date >>/tmp/connection.count
  db2 list application |wc -l >>/tmp/connection.count
  
  The /tmp/connection.count output file will allow to map out the
  date/time and number of database connections.
  
  
  The example below shows a steady increase of sessions
  resulting in a server restart being required:
  
  Sat Aug 21 19:15:00 CEST 2021
     9509
  
  Sat Aug 21 19:20:00 CEST 2021
     9517
  
  Sat Aug 21 19:25:00 CEST 2021
     9544
  
  Sat Aug 21 19:30:00 CEST 2021
     9546
  
  This APAR fixes the ANR0102E replication error and session
  growth.
  
  IBM Spectrum Protect Server Version Affected:
  
  Version 8.1.12  and above on all supported platforms
  
  | MDVREGR 8.1.12.-TIV_5698MSV |
  
  Additional Keywords:
  
  TSM SERVER Spectrum Protect TS006548227 NODE REPLICATION
  ANR0102E ANR9999D_0941050150
  

Local fix

• Restart the  IBM  Spectrum Protect server process/service to
  clear down the database connections.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect server customers using node             *
  * replication.                                                 *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 8.1.13. Note that this is     *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:   AIX, Linux, Windows
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels