IBM Support

IT27943: Deadlatch between an online backup, prefetchers and an agent doing a write to the same table space

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • A deadlatch might happen when an online backup is holding the
    latch SQLO_LT_SQLB_POOL_CB__readLotch in S mode and waiting on
    free prefetcher requests in the following stack:
    
    
    <StackTrace>
    -----FUNC-ADDR---- ------FUNCTION + OFFSET------
    0x00007FCD0390CCA6
    _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
    0x0356
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0390C8EB ossDumpStackTraceV98 + 0x002b
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD03908AD7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
    0x00d7
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0B170187 sqlo_trce + 0x03c7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2DBC0B sqloDumpDiagInfoHandler + 0x020b
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E71680 address: 0x00007FCD12E71680 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x000000000000F680 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD029172A3 msgrcv + 0x0033
                    (/lib64/libc.so.6)
    0x00007FCD0B27C259 sqloCSemP + 0x0279
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DB6D14
    _Z26sqlbPFFindAndLatchFreelistiPP15SQLB_pfFreelistP12SQLB_GLOBAL
    S + 0x04b4
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DBA0F8
    _Z29sqlbPFGetMultipleFreeRequestsjiPP14SQLB_pfRequestPmP12SQLB_G
    LOBALS14SQLB_pfReqTypeP16SQLB_OBJECT_DESCt + 0x0068
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DB87DA
    _Z25sqlbpfParallelDirectReadVP14SQLB_pfParIoCbP17SQLB_DIRECT_IO_
    CB + 0x18aa
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05BE2FDD
    _Z19sqlbDirectReadBlockP19SQLB_DIRECT_READ_CBP14SQLB_pfParIoCbjP
    jS3_ + 0x05dd
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09C5B295 _Z12sqlubReadDMSP14SQLU_BUFMAN_CB + 0x0485
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09C59C14 _Z18sqlubBMProcessDataP14SQLU_BUFMAN_CB +
    0x0304
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09C5915F _Z8sqlubbufP8sqeAgent + 0x051f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09476320 _Z26sqleSubCoordProcessRequestP8sqeAgent +
    0x0150
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD094931AE _ZN8sqeAgent6RunEDUEv + 0x05ae
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0CDB321E _ZN9sqzEDUObj9EDUDriverEv + 0x01be
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2D96BA sqloEDUEntry + 0x057a
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E69DD5 address: 0x00007FCD12E69DD5 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x0000000000007DD5 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02915B3D clone + 0x006d
                    (/lib64/libc.so.6)
    </StackTrace>
    
    
    An agent trying to make some change to the same table space is
    waiting behind the online backup on the latch
    SQLO_LT_SQLB_POOL_CB__readLotch in X mode, a sample stack is as
    follow.  Note it is not always the same stack because it depends
    on what writing operation this agent is doing to the table
    space.
    
    <StackTrace>
    -----FUNC-ADDR---- ------FUNCTION + OFFSET------
    
    0x00007FCD0390CCA6
    _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
    0x0356
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0390C8EB ossDumpStackTraceV98 + 0x002b
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD03908AD7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
    0x00d7
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0B170187 sqlo_trce + 0x03c7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2DBC0B sqloDumpDiagInfoHandler + 0x020b
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E71680 address: 0x00007FCD12E71680 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x000000000000F680 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02917337 semop + 0x0007
                    (/lib64/libc.so.6)
    0x00007FCD0B16E3A2 _ZN17SQLO_SLATCH_CAS6418getConflictComplexEm
    + 0x07e2
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B16DBBC _ZN17SQLO_SLATCH_CAS6411getConflictEm +
    0x007c
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05363F6F _ZN12sqlpValLotch12getLatchOnlyEmPKcm +
    0x016f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05454E18
    _Z21sqlbLatchPoolR_inlineP12SQLB_POOL_CBibiPKc + 0x00b8
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05BC1799 _Z25sqlbLatchPoolForReadWriteP12SQLB_POOL_CB
    + 0x0029
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05DEC71E
    _Z13sqlbUpdateMRTtP9SQLP_LSN8PmP12SQLB_GLOBALS + 0x00ee
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0605E6D3
    _Z8sqldmpndP8sqeAgentiPcP9SQLP_LSN8PmP15SQLD_RECOV_INFO + 0x0c23
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B3D8518 _Z8sqlptpplP8sqeAgentiPb + 0x0668
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B3BBEBF _Z8sqlpxcm1P8sqeAgentP15SQLXA_CALL_INFOi +
    0x042f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B96447F
    _Z12sqlrrcom_dpsP8sqlrr_cbiiP15SQLXA_CALL_INFO + 0x165f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B961FE0 _Z8sqlrrcomP8sqlrr_cbii + 0x0550
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B93A176 _Z12sqlrr_commitP14db2UCinterface + 0x0136
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD09A8E1E8
    _Z16sqljs_ddm_rdbcmmP14db2UCinterfaceP13sqljDDMObject + 0x0228
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    ...
    </StackTrace>
    
    At the same time, prefetchers are waiting behind the writer
    agent on the latch SQLO_LT_SQLB_POOL_CB__readLotch in S mode in
    the following stack:
    
    <StackTrace>
    -----FUNC-ADDR---- ------FUNCTION + OFFSET------
    0x00007FCD0390CCA6
    _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
    0x0356
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0390C8EB ossDumpStackTraceV98 + 0x002b
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD03908AD7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
    0x00d7
                    (/home/db2mcys/sqllib/lib64/libdb2osse.so.1)
    0x00007FCD0B170187 sqlo_trce + 0x03c7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2DBC0B sqloDumpDiagInfoHandler + 0x020b
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E71680 address: 0x00007FCD12E71680 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x000000000000F680 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02917337 semop + 0x0007
                    (/lib64/libc.so.6)
    0x00007FCD0B16E3A2 _ZN17SQLO_SLATCH_CAS6418getConflictComplexEm
    + 0x07e2
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B16DBBC _ZN17SQLO_SLATCH_CAS6411getConflictEm +
    0x007c
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05363F6F _ZN12sqlpValLotch12getLatchOnlyEmPKcm +
    0x016f
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05454E18
    _Z21sqlbLatchPoolR_inlineP12SQLB_POOL_CBibiPKc + 0x00b8
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B9824B
    _Z25sqlbReadAndReleaseBuffersP16SQLB_OBJECT_DESCjmmmP11SQLB_pfMi
    sc + 0x02bb
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B9B9F7
    _Z22sqlbServiceListRequestP11SQLB_pfMiscP16SQLB_OBJECT_DESCPjj +
    0x04b7
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B95C17
    _Z26sqlbPFPrefetcherEntryPointP16sqbPrefetcherEdu + 0x1227
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD05B94964 _ZN16sqbPrefetcherEdu6RunEDUEv + 0x0024
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0CDB321E _ZN9sqzEDUObj9EDUDriverEv + 0x01be
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD0B2D96BA sqloEDUEntry + 0x057a
                    (/home/db2mcys/sqllib/lib64/libdb2e.so.1)
    0x00007FCD12E69DD5 address: 0x00007FCD12E69DD5 ; dladdress:
    0x00007FCD12E62000 ; offset in lib: 0x0000000000007DD5 ;
                    (/lib64/libpthread.so.0)
    0x00007FCD02915B3D clone + 0x006d
                    (/lib64/libc.so.6)
    </StackTrace>
    

Local fix

  • db2set DB2BPVARS=<path to config file>
    in the config file, have this line:
    NOREADVBCKUP=1
    
    Note that this might have performance impact on the backup
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * ALL                                                          *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See Error Description                                        *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Upgrade to DB2 Version 11.1 Module 4 Fix Pack 5              *
    ****************************************************************
    

Problem conclusion

  • Problem was first fixed in DB2 Version 11.1 Module 4 Fix Pack 5
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT27943

  • Reported component name

    DB2 FOR LUW

  • Reported component ID

    DB2FORLUW

  • Reported release

    B10

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2019-01-31

  • Closed date

    2019-10-31

  • Last modified date

    2019-10-31

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    IT30784

Fix information

  • Fixed component name

    DB2 FOR LUW

  • Fixed component ID

    DB2FORLUW

Applicable component levels

  • RB10 PSN

       UP

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSEPGG","label":"DB2 for Linux, UNIX and Windows"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"11.1","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
31 October 2019