IT32619: DB2START FAILS DUE TO ENFORCEMENT OF MAXIMUM SEMAPHORE ARRAYS ONLINUX KERNELS 3.10.0-1127.EL7 AND 4.18.0-147.EL8_1 AND NEWER

APAR status

• Closed as program error.

Error description

• A db2start or other operations may fail due to new kernel
  enforcement of maximum semaphore arrays (SEMMNI) and message
  queue resources (MSGMNI)starting with RHEL 7.8
  kernel version 3.10.0-1127.el7.x86_64, or RHEL 8.1 kernel
  version 4.18.0-147.el8_1.x86_64.
  
  Starting with RHEL 7.8 kernel version 3.10.0-1127.el7.x86_64, or
  RHEL 8.1 kernel version 4.18.0-147.el8_1.x86_64, the maximum
  number of configurable semaphore arrays is now enforced to
  32768.
  This change is kernel enforcement behaviour is described in the
  following Redhat Knowledgebase article:
  https://access.redhat.com/solutions/4968021.
  
  On these new kernel versions, a db2start operation which
  succeeded in a previous kernel version, may now fail because it
  is not able to configure greater then 32768 semaphore arrays and
  message queues.
  
  Semaphores are used for database inter-process/thread
  communication, and thus databases with workloads requiring a
  significant number of agents/EDUs will require more semaphores.
  
  When the configured maximum number of kernel semaphores arrays
  (kernel.sem SEMMNI) or message queues (kernel.msgmni) is less
  then the minimum value required by
  Db2 (as documented here:
  https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ib
  m.db2.luw.qb.server.doc/doc/c0057140.html), then db2start will
  attempt to automatically increase the maximum number of
  semaphore arrays to 256*(size of RAM in GB). Thus when greater
  then 128GB of RAM exists, then Db2 will not be able to configure
  greater then 32768 semaphore arrays.
  
  
  db2start will then proceed with the currently configured kernel
  maximum number of semaphore arrays (kernel.sem SEMMNI) in place,
  however if this value is low (such as the kernel default 128),
  then db2start may fail to allocate sufficient semaphores for
  database startup, and the following entry will appear in the
  db2diag.log:
  
  2020-04-14-01.23.06.849143-420 E42143E400            LEVEL:
  Error (OS)
  PID     : <pid>                TID : <tid> PROC : db2sysc 0
  INSTANCE: <instname>              NODE : 000
  HOSTNAME: <hostname>
  FUNCTION: DB2 UDB, oper system services,
  sqlo_waitlist::initialize, probe:10
  MESSAGE : ZRC=0x8300001C=-2097151972
  CALLED  : OS, -, semget                           OSERR: ENOSPC
  (28)
  
  2020-04-14-11.16.14.698368-420 E72551E480 LEVEL: Severe
  PID : <pid>  TID : <tid> PROC : db2sysc 0
  INSTANCE: <instname> NODE : 000
  HOSTNAME: <hostname>
  FUNCTION: DB2 UDB, oper system services, sqloEDUEntry, probe:10
  CALLED : DB2 UDB, oper system services, sqloGetShrEDUWaitElem
  RETCODE : ZRC=0x850F0081=-2062614399=SQLO_SSEM_EXCEED_MAX
  "Requesting too many semaphores"
  DIA8336C Requested too many semaphores.
  
  Alternately, if the currently configured kernel maximum number
  of semaphore arrays (kernel.sem SEMMNI) is sufficient for
  db2start to succeed,  db2 may fail later on if the database
  workload necessitates an increase in the number agents/EDUs and
  more semaphore arrays are required and exceed the maximum.
  
  You can use 'ipcs -l' to examine the 'max number of arrays':
  
  $  ipcs -l
  ------ Semaphore Limits --------
  max number of arrays = 128 /* SEMMNI */
  max semaphores per array = 250 /* SEMMSL */
  max semaphores system wide = 256000 /* SEMMNS */
  max ops per semop call = 32 /* SEMOPM */
  semaphore max value = 32767
  
  ------ Messages Limits --------
  max queues system wide = 3200 /* MSGMNI */
  
  or check the file /etc/sysctl.conf
  kernel.shmmni=786432
  #kernel.sem=<SEMMSL> <SEMMNS> <SEMOPM> <SEMMNI>
  kernel.sem=250 256000 32 128
  kernel.msgmni=3200
  
  
  This fix configures DB2 to use up to 32768 for semaphore arrays
  and message queues.
  
  Additional Reference:
  Are there any upper limits for msgmax, msgmni and msgmnb in the
  System V IPC message queue?
  https://access.redhat.com/articles/15423
  

Local fix

• Increase the maximum number of kernel semaphore arrays
  (kernel.sem SEMMNI) and message queues (kernel.msgmni) to 32768.
  The maximum value of 32768 is generally
  sufficient.  
  Instructions on how to update kernel parameters can be found
  here:
  (https://access.redhat.com/documentation/en-us/red_hat_enterpris
  e_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_fo
  r_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guid
  e-setting_semaphores-setting_semaphore_parameters). 
  
  OR
  
  Install a db2 fixpack inclusive of APAR fix IT32619.  As of time
  of this writing, a fixpack inclusive of this fix is not yet
  available, but consult the APAR link to determine availability.
  IT32619 limits both values to 32768. In theory, DB2 could
  encounter maximum limit when trying to self adjust kernel
  parameters beyond 32768, thus for additional flexibility
  enabling ipcmni_extend kernel is the alternative.
  
  OR
  Enable the ipcmni_extend kernel parameter to allow for more then
  32768 semaphore arrays and message queues (as documented in the
  Redhat
  Knowledgebase article
  (https://access.redhat.com/solutions/4968021).
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Large memory configurations (multiple terabytes)             *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade server to v11.1 M4 FP6                               *
  ****************************************************************
  

Problem conclusion

• First fixed in v11.1 M4 FP6
  

Temporary fix

• See LOCAL FIX
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  DB2 FOR LUW

• Fixed component ID

  DB2FORLUW

Applicable component levels