IT28505: FAILURES DUE TO THE SERVER RUNNING OUT OF DATABASE CONNECTIONS

8.1.9.000-IBM-SPSRV-WindowsX64
8.1.9.000-IBM-SPSRV-Linuxx86_64
8.1.9.000-IBM-SPOC-WindowsX64
8.1.9.000-IBM-SPCMS-WindowsX64
8.1.9.000-IBM-SPSRV-Linuxs390x
8.1.9.000-IBM-SPCMS-WindowsI32
8.1.9.000-IBM-SPOC-Linuxx86_64
8.1.9.000-IBM-SPCMS-Linuxx86_64
8.1.9.000-IBM-SPSRV-Linuxppc64le
8.1.9.000-IBM-SPOC-Linuxs390x
8.1.9.000-IBM-SPOC-LinuxPPC64le
8.1.9.000-IBM-SPSRV-AIX
8.1.9.000-IBM-SPOC-AIX

APAR status

• Closed as program error.

Error description

• MAXSESSION is set above 1000.   Query Session has over 1000
  Sessions in a "Run" Session State.  In the case reported, there
  were 1089 Run state sessions with many other sessions in IdleW,
  RecvW, and SendW.
  
  Key actlog error:
  ANR9999D_0941050150 InitConnectionPool(dbiconn.c:1686)
  Thread<45746>: All connection pools are full. (SESSION: xxxx)
  
  If the above conditions are seen, then this APAR is impacting
  the server.
  
  Other errors that may be reported in the dsmffdc.log:
  [ FFDC_GENERAL_SERVER_ERROR ]: (smexec.c:8335) Unexpected error
  (9999) for session xx(node_name)
  [ FFDC_GENERAL_SERVER_ERROR ]: (sdcreate.c:1682) Exiting with
  bad rc=9991 for bfId xxxxx.
  [ FFDC_ERROR_FROM_DB2 ]: (tbcli.c:2549) Error inserting into
  table Backup.Objects for row 0 containing key(s): NODEID: xx,
  [ FFDC_GENERAL_SERVER_ERROR ]: (sdcntr.c:10540) Error 9991
  getting container in directory NULL.
  [ FFDC_GENERAL_SERVER_ERROR ]: (sdtxn.c:2383) Txn 0:xxx ran for
  10961 seconds exceeding max runtime of 180
  
  L2 Diagnostics:
  L2 provides steps to add the SHOW DBCONN to the servermon
  script.  Look for:
  "Total number of connections"  This is over 9000
  
  IBM Spectrum Protect Versions Affected:   7.1. and above on all
  platforms.
  
  Initial Impact: Low
  Additional Keywords:  db anr9999d Cloud cache deletion
  Case:  TS001826772
  

Local fix

• Set the MAXSESSION to 1000 or below.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager server users.                     *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 8.1.9. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, HP-UX, Solaris,
  Linux, and Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels