IBM Support

IT32619: DB2START FAILS DUE TO ENFORCEMENT OF MAXIMUM SEMAPHORE ARRAYS ONLINUX KERNELS 3.10.0-1127.EL7 AND 4.18.0-147.EL8_1 AND NEWER

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • A db2start or other operations may fail due to new kernel
    enforcement of maximum semaphore arrays (SEMMNI) and message
    queue resources (MSGMNI)starting with RHEL 7.8
    kernel version 3.10.0-1127.el7.x86_64, or RHEL 8.1 kernel
    version 4.18.0-147.el8_1.x86_64.
    
    Starting with RHEL 7.8 kernel version 3.10.0-1127.el7.x86_64, or
    RHEL 8.1 kernel version 4.18.0-147.el8_1.x86_64, the maximum
    number of configurable semaphore arrays is now enforced to
    32768.
    This change is kernel enforcement behaviour is described in the
    following Redhat Knowledgebase article:
    https://access.redhat.com/solutions/4968021.
    
    On these new kernel versions, a db2start operation which
    succeeded in a previous kernel version, may now fail because it
    is not able to configure greater then 32768 semaphore arrays and
    message queues.
    
    Semaphores are used for database inter-process/thread
    communication, and thus databases with workloads requiring a
    significant number of agents/EDUs will require more semaphores.
    
    When the configured maximum number of kernel semaphores arrays
    (kernel.sem SEMMNI) or message queues (kernel.msgmni) is less
    then the minimum value required by
    Db2 (as documented here:
    https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ib
    m.db2.luw.qb.server.doc/doc/c0057140.html), then db2start will
    attempt to automatically increase the maximum number of
    semaphore arrays to 256*(size of RAM in GB). Thus when greater
    then 128GB of RAM exists, then Db2 will not be able to configure
    greater then 32768 semaphore arrays.
    
    
    db2start will then proceed with the currently configured kernel
    maximum number of semaphore arrays (kernel.sem SEMMNI) in place,
    however if this value is low (such as the kernel default 128),
    then db2start may fail to allocate sufficient semaphores for
    database startup, and the following entry will appear in the
    db2diag.log:
    
    2020-04-14-01.23.06.849143-420 E42143E400            LEVEL:
    Error (OS)
    PID     : <pid>                TID : <tid> PROC : db2sysc 0
    INSTANCE: <instname>              NODE : 000
    HOSTNAME: <hostname>
    FUNCTION: DB2 UDB, oper system services,
    sqlo_waitlist::initialize, probe:10
    MESSAGE : ZRC=0x8300001C=-2097151972
    CALLED  : OS, -, semget                           OSERR: ENOSPC
    (28)
    
    2020-04-14-11.16.14.698368-420 E72551E480 LEVEL: Severe
    PID : <pid>  TID : <tid> PROC : db2sysc 0
    INSTANCE: <instname> NODE : 000
    HOSTNAME: <hostname>
    FUNCTION: DB2 UDB, oper system services, sqloEDUEntry, probe:10
    CALLED : DB2 UDB, oper system services, sqloGetShrEDUWaitElem
    RETCODE : ZRC=0x850F0081=-2062614399=SQLO_SSEM_EXCEED_MAX
    "Requesting too many semaphores"
    DIA8336C Requested too many semaphores.
    
    Alternately, if the currently configured kernel maximum number
    of semaphore arrays (kernel.sem SEMMNI) is sufficient for
    db2start to succeed,  db2 may fail later on if the database
    workload necessitates an increase in the number agents/EDUs and
    more semaphore arrays are required and exceed the maximum.
    
    You can use 'ipcs -l' to examine the 'max number of arrays':
    
    $  ipcs -l
    ------ Semaphore Limits --------
    max number of arrays = 128 /* SEMMNI */
    max semaphores per array = 250 /* SEMMSL */
    max semaphores system wide = 256000 /* SEMMNS */
    max ops per semop call = 32 /* SEMOPM */
    semaphore max value = 32767
    
    ------ Messages Limits --------
    max queues system wide = 3200 /* MSGMNI */
    
    or check the file /etc/sysctl.conf
    kernel.shmmni=786432
    #kernel.sem=<SEMMSL> <SEMMNS> <SEMOPM> <SEMMNI>
    kernel.sem=250 256000 32 128
    kernel.msgmni=3200
    
    
    This fix configures DB2 to use up to 32768 for semaphore arrays
    and message queues.
    
    Additional Reference:
    Are there any upper limits for msgmax, msgmni and msgmnb in the
    System V IPC message queue?
    https://access.redhat.com/articles/15423
    

Local fix

  • Increase the maximum number of kernel semaphore arrays
    (kernel.sem SEMMNI) and message queues (kernel.msgmni) to 32768.
    The maximum value of 32768 is generally
    sufficient.  
    Instructions on how to update kernel parameters can be found
    here:
    (https://access.redhat.com/documentation/en-us/red_hat_enterpris
    e_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_fo
    r_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guid
    e-setting_semaphores-setting_semaphore_parameters). 
    
    OR
    
    Install a db2 fixpack inclusive of APAR fix IT32619.  As of time
    of this writing, a fixpack inclusive of this fix is not yet
    available, but consult the APAR link to determine availability.
    IT32619 limits both values to 32768. In theory, DB2 could
    encounter maximum limit when trying to self adjust kernel
    parameters beyond 32768, thus for additional flexibility
    enabling ipcmni_extend kernel is the alternative.
    
    OR
    Enable the ipcmni_extend kernel parameter to allow for more then
    32768 semaphore arrays and message queues (as documented in the
    Redhat
    Knowledgebase article
    (https://access.redhat.com/solutions/4968021).
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * Large memory configurations (multiple terabytes)             *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See Error Description                                        *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Upgrade server to v11.1 M4 FP6                               *
    ****************************************************************
    

Problem conclusion

  • First fixed in v11.1 M4 FP6
    

Temporary fix

  • See LOCAL FIX
    

Comments

APAR Information

  • APAR number

    IT32619

  • Reported component name

    DB2 FOR LUW

  • Reported component ID

    DB2FORLUW

  • Reported release

    B10

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2020-04-21

  • Closed date

    2021-03-19

  • Last modified date

    2021-03-19

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    DB2 FOR LUW

  • Fixed component ID

    DB2FORLUW

Applicable component levels

[{"Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEPGG","label":"Db2 for Linux, UNIX and Windows"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"11.1"}]

Document Information

Modified date:
20 March 2021