IT41189: ANR9999D_0941050150 INITCONNECTIONPOOL ALL CONNECTION POOLS ARE FULL CAUSED BY OPERATIONS CENTER STORAGE RULES PANELS

8.1.17.000-IBM-SPOC-WindowsX64
8.1.17.000-IBM-SPOC-Linuxx86_64
8.1.17.000-IBM-SPOC-Linuxs390x
8.1.17.000-IBM-SPOC-LinuxPPC64le
8.1.17.000-IBM-SPOC-AIX
8.1.17.000-IBM-SPSRV-WindowsX64
8.1.17.000-IBM-SPSRV-Linuxx86_64
8.1.17.000-IBM-SPSRV-Linuxs390x
8.1.17.000-IBM-SPSRV-Linuxppc64le
8.1.17.000-IBM-SPSRV-AIX
8.1.16.100-IBM-SPSRV-WindowsX64
8.1.16.100-IBM-SPSRV-Linuxx86_64
8.1.16.100-IBM-SPSRV-Linuxs390x
8.1.16.100-IBM-SPSRV-Linuxppc64le
8.1.16.100-IBM-SPSRV-AIX
8.1.15.100-IBM-SPSRV-WindowsX64
8.1.15.100-IBM-SPSRV-Linuxx86_64
8.1.15.100-IBM-SPSRV-Linuxs390x
8.1.15.100-IBM-SPSRV-Linuxppc64le
8.1.15.100-IBM-SPSRV-AIX

APAR status

• Closed as program error.

Error description

• After viewing the Storage Rules page in the Operations Center,
  the following error may happen and cause server instability:
  
  ANR9999D_0941050150 InitConnectionPool(dbiconn.c:1709)
  Thread<xx>: All connection pools are full
  
  Other error messages, process failures, or performance issues
  may appear after that. This happens because there are some
  orphaned transactions created when navigating in the Operations
  Center Storage Rules panels, and each orphan uses and holds onto
  one database connection. The IBM Spectrum Protect server has a
  limit of 9600 concurrent database connections at any given time.
  
  
  
  The following commands can be used to monitor connections
  usage over time:
  
  date >>/tmp/connection.count
  db2 list application |wc -l >>/tmp/connection.count
  
  The /tmp/connection.count output file will show the date/time
  and number of database connections.
  If the number of database connections reaches 9600, then the
  ANR9999D_0941050150 InitConnectionPool error appears in the
  activity log.
  
  In the servermon data collection file *-10min-show.* the SHOW
  TXNTABLE output will show a lot of these transactions:
  
  Tsn=0:873444608, Resurrected=False, InFlight=True,
  Distributed=False, Persistent=True, Addr 0x7fb4e46cb018
   Start ThreadId=863333, Timestamp=16/05/22 11:00:18,
  Creator=scclass.c(6778)
   Last known in use by ThreadId=863333
   Participants=1, summaryVote=ReadOnly
   EndInFlight False, endThreadId 0, tmidx 0, processBatchCount 0,
  mustAbort False.
     Participant DB: voteReceived=False, ackReceived=False
       DB: Txn 0x7fb4e44438b8, ReadOnly(YES),
  connP=0x7fb4e4070608, applHandle=44730, openTbls=1:
       DB: --> OpenP=0x7fb4e40d5378 for table=SC.Stg.Subrules.
  
  The keys are the ?Creator=scclass.c? and
  ?table=SC.Stg.Subrules?.
  The ThreadId number (in this case ?863333?) is not found in the
  same servermon output file in the SHOW THREAD output.
  
  This means that these transactions are orphaned and will never
  release their database connections.
  
  
  IBM Spectrum Protect Versions Affected: Version 8.1 on all
   supported platforms
  
  Additional Keywords: TS009363016, stgrule, orphan, OC
  

Local fix

• Restart the IBM Spectrum Protect server to clear the orphaned
  transactions
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.15.100, 8.1.16.100 and   *
  * 8.1.17. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels