IBM Support

IT27943: Deadlatch between an online backup, prefetchers and an agent doing a write to the same table space

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A deadlatch might happen when an online backup is holding the
    latch SQLO_LT_SQLB_POOL_CB__readLotch in S mode and waiting on
    free prefetcher requests in the following stack:
    
    
    <StackTrace>
    -----FUNC-ADDR---- ------FUNCTION + OFFSET------
    0x00007FCD0390CCA6
    _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
    0x0356
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0390C8EB ossDumpStackTraceV98 + 0x002b
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD03908AD7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
    0x00d7
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0B170187 sqlo_trce + 0x03c7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2DBC0B sqloDumpDiagInfoHandler + 0x020b
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E71680 address: 0x00007FCD12E71680 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x000000000000F680 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD029172A3 msgrcv + 0x0033
                    (/lib64/libc.so.6)
    0x00007FCD0B27C259 sqloCSemP + 0x0279
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DB6D14
    _Z26sqlbPFFindAndLatchFreelistiPP15SQLB_pfFreelistP12SQLB_GLOBAL
    S + 0x04b4
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DBA0F8
    _Z29sqlbPFGetMultipleFreeRequestsjiPP14SQLB_pfRequestPmP12SQLB_G
    LOBALS14SQLB_pfReqTypeP16SQLB_OBJECT_DESCt + 0x0068
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DB87DA
    _Z25sqlbpfParallelDirectReadVP14SQLB_pfParIoCbP17SQLB_DIRECT_IO_
    CB + 0x18aa
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05BE2FDD
    _Z19sqlbDirectReadBlockP19SQLB_DIRECT_READ_CBP14SQLB_pfParIoCbjP
    jS3_ + 0x05dd
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09C5B295 _Z12sqlubReadDMSP14SQLU_BUFMAN_CB + 0x0485
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09C59C14 _Z18sqlubBMProcessDataP14SQLU_BUFMAN_CB +
    0x0304
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09C5915F _Z8sqlubbufP8sqeAgent + 0x051f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09476320 _Z26sqleSubCoordProcessRequestP8sqeAgent +
    0x0150
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD094931AE _ZN8sqeAgent6RunEDUEv + 0x05ae
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0CDB321E _ZN9sqzEDUObj9EDUDriverEv + 0x01be
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2D96BA sqloEDUEntry + 0x057a
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E69DD5 address: 0x00007FCD12E69DD5 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x0000000000007DD5 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02915B3D clone + 0x006d
                    (/lib64/libc.so.6)
    </StackTrace>
    
    
    An agent trying to make some change to the same table space is
    waiting behind the online backup on the latch
    SQLO_LT_SQLB_POOL_CB__readLotch in X mode, a sample stack is as
    follow.  Note it is not always the same stack because it depends
    on what writing operation this agent is doing to the table
    space.
    
    <StackTrace>
    -----FUNC-ADDR---- ------FUNCTION + OFFSET------
    
    0x00007FCD0390CCA6
    _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
    0x0356
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0390C8EB ossDumpStackTraceV98 + 0x002b
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD03908AD7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
    0x00d7
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0B170187 sqlo_trce + 0x03c7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2DBC0B sqloDumpDiagInfoHandler + 0x020b
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E71680 address: 0x00007FCD12E71680 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x000000000000F680 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02917337 semop + 0x0007
                    (/lib64/libc.so.6)
    0x00007FCD0B16E3A2 _ZN17SQLO_SLATCH_CAS6418getConflictComplexEm
    + 0x07e2
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B16DBBC _ZN17SQLO_SLATCH_CAS6411getConflictEm +
    0x007c
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05363F6F _ZN12sqlpValLotch12getLatchOnlyEmPKcm +
    0x016f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05454E18
    _Z21sqlbLatchPoolR_inlineP12SQLB_POOL_CBibiPKc + 0x00b8
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05BC1799 _Z25sqlbLatchPoolForReadWriteP12SQLB_POOL_CB
    + 0x0029
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DEC71E
    _Z13sqlbUpdateMRTtP9SQLP_LSN8PmP12SQLB_GLOBALS + 0x00ee
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0605E6D3
    _Z8sqldmpndP8sqeAgentiPcP9SQLP_LSN8PmP15SQLD_RECOV_INFO + 0x0c23
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B3D8518 _Z8sqlptpplP8sqeAgentiPb + 0x0668
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B3BBEBF _Z8sqlpxcm1P8sqeAgentP15SQLXA_CALL_INFOi +
    0x042f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B96447F
    _Z12sqlrrcom_dpsP8sqlrr_cbiiP15SQLXA_CALL_INFO + 0x165f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B961FE0 _Z8sqlrrcomP8sqlrr_cbii + 0x0550
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B93A176 _Z12sqlrr_commitP14db2UCinterface + 0x0136
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09A8E1E8
    _Z16sqljs_ddm_rdbcmmP14db2UCinterfaceP13sqljDDMObject + 0x0228
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    ...
    </StackTrace>
    
    At the same time, prefetchers are waiting behind the writer
    agent on the latch SQLO_LT_SQLB_POOL_CB__readLotch in S mode in
    the following stack:
    
    <StackTrace>
    -----FUNC-ADDR---- ------FUNCTION + OFFSET------
    0x00007FCD0390CCA6
    _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
    0x0356
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0390C8EB ossDumpStackTraceV98 + 0x002b
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD03908AD7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
    0x00d7
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0B170187 sqlo_trce + 0x03c7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2DBC0B sqloDumpDiagInfoHandler + 0x020b
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E71680 address: 0x00007FCD12E71680 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x000000000000F680 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02917337 semop + 0x0007
                    (/lib64/libc.so.6)
    0x00007FCD0B16E3A2 _ZN17SQLO_SLATCH_CAS6418getConflictComplexEm
    + 0x07e2
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B16DBBC _ZN17SQLO_SLATCH_CAS6411getConflictEm +
    0x007c
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05363F6F _ZN12sqlpValLotch12getLatchOnlyEmPKcm +
    0x016f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05454E18
    _Z21sqlbLatchPoolR_inlineP12SQLB_POOL_CBibiPKc + 0x00b8
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B9824B
    _Z25sqlbReadAndReleaseBuffersP16SQLB_OBJECT_DESCjmmmP11SQLB_pfMi
    sc + 0x02bb
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B9B9F7
    _Z22sqlbServiceListRequestP11SQLB_pfMiscP16SQLB_OBJECT_DESCPjj +
    0x04b7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B95C17
    _Z26sqlbPFPrefetcherEntryPointP16sqbPrefetcherEdu + 0x1227
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B94964 _ZN16sqbPrefetcherEdu6RunEDUEv + 0x0024
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0CDB321E _ZN9sqzEDUObj9EDUDriverEv + 0x01be
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2D96BA sqloEDUEntry + 0x057a
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E69DD5 address: 0x00007FCD12E69DD5 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x0000000000007DD5 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02915B3D clone + 0x006d
                    (/lib64/libc.so.6)
    </StackTrace>
    

Local fix

  • db2set DB2BPVARS=<path to config file>
    in the config file, have this line:
    NOREADVBCKUP=1
    
    Note that this might have performance impact on the backup
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * ALL                                                          *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See Error Description                                        *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Upgrade to DB2 Version 11.1 Module 4 Fix Pack 5              *
    ****************************************************************
    

Problem conclusion

  • Problem was first fixed in DB2 Version 11.1 Module 4 Fix Pack 5
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT27943

  • Reported component name

    DB2 FOR LUW

  • Reported component ID

    DB2FORLUW

  • Reported release

    B10

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2019-01-31

  • Closed date

    2019-10-31

  • Last modified date

    2019-10-31

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    IT30784

Fix information

  • Fixed component name

    DB2 FOR LUW

  • Fixed component ID

    DB2FORLUW

Applicable component levels

  • RB10 PSN

       UP

[{"Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SSEPGG","label":"DB2 for Linux- UNIX and Windows"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"11.1"}]

Document Information

Modified date:
03 September 2021