IT28756: Db2 11.1 Mod4 FP4 (including iFix 001) might hang, crash or fail with SQL0958C due to EMFILE error (Too many open files) on AIX

APAR status

• Closed as program error.

Error description

• Db2 11.1 Mod4 FP4 will leak 2 file descriptors on each
  database activation. In the environments where databases are
  frequently deactivated and activated this might lead to db2sysc
  process running out of file descriptors and EMFILE error.
  Depending on the code path which hits the error first, this
  might be manifested as hang, crash or an error returned to the
  application (SQL0958C).
  When error is hit, an entry to similar to this one will be
  logged to db2diag.log
  
  FUNCTION: DB2 UDB, oper system services, sqloopenp, probe:80
  MESSAGE : ZRC=0x850F0006=-2062614522=SQLO_FHNL "TOO MANY OPEN
  FILES"
            DIA8306C Too many files were opened.
  CALLED  : OS, -, open
  OSERR   : EMFILE (24) "Too many open files"
  
  and running "procfiles" on db2sysc will show that files opened
  by the process are using very high file descriptor IDs (close to
  65535), e.g.:
  
  $ procfiles -n pid_of_db2sysc | tail
     12: S_IFREG mode:0640 dev:10,8 ino:5814 uid:1111 gid:204
  rdev:0,0
        O_RDWR | O_DSYNC size:24576
  name:/home/db2v111/db2v111/NODE0000/SQL00001/MEMBER0000/SQLOGCTL
  .LFH.2
     13: S_IFREG mode:0640 dev:10,8 ino:5815 uid:1111 gid:204
  rdev:0,0
        O_RDWR | O_DSYNC size:8192
  name:/home/db2v111/db2v111/NODE0000/SQL00001/MEMBER0000/SQLOGMIR
  .LFH
     65530: S_IFREG mode:0400 dev:10,8 ino:5823 uid:1111 gid:204
  rdev:0,0
        O_RDONLY | O_DIRECT size:4104192
  name:/home/db2v111/db2v111/NODE0000/SQL00001/LOGSTREAM0000/S0000
  000.LOG
     65531: S_IFREG mode:0400 dev:10,8 ino:5825 uid:1111 gid:204
  rdev:0,0
        O_RDONLY | O_DIRECT size:4104192
  name:/home/db2v111/db2v111/NODE0000/SQL00001/LOGSTREAM0000/S0000
  001.LOG
     65532: S_IFREG mode:0400 dev:10,8 ino:5826 uid:1111 gid:204
  rdev:0,0
        O_RDONLY | O_DIRECT size:4104192
  name:/home/db2v111/db2v111/NODE0000/SQL00001/LOGSTREAM0000/S0000
  002.LOG
  
  "lsof" tool will indicate multiple descriptors opened with name
  "file struct, ty=0x5"
  
  To clear the problem, instance restart is required
  

Local fix

• Avoid frequent database activation and deactivation by
  explicitly activating the database:
  $ db2 activate database <db_name>
  
  Increasing the 'nofiles' limit to reduce the likelihood of
  hitting the apar.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * ALL                                                          *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade to Db2 11.1 Mod 4 Fixpack 5 or higher                *
  ****************************************************************
  

Problem conclusion

• First fixed in Db2 11.1 Mod 4 Fixpack 5
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  DB2 FOR LUW

• Fixed component ID

  DB2FORLUW

Applicable component levels

• RB10 PSN

     UP