IT24299: DB2 CRASH IN DPF + SMP ENVIRONMENT AFTER INTERNAL INTERRUPT (SQL0952N)

Media_IM_5020603_6
Media_IM_5020603_6
iSeries_Media_5020603_6.zip
iSeries_Media_5020603_6

APAR status

• Closed as program error.

Error description

• IN DPF+SMP environment traps can happen with the following
  stacks:
  
  sqloCrashOnCriticalMemoryValidationFailure
  diagnoseMemoryCorruptionAndCrash
  sqloDiagnoseFreeBlockFailure
  sqlofmblk
  sqlristm
  sqlra_sqlW_mem_free_sibling_list
  
  db2diag.log Could have the following :
  
  
  FUNCTION: DB2 UDB, base sys utilities,
  sqeAgentServices::RequestAgentForAppl, probe:1808
  MESSAGE : Appl failed to start up on this node. Interrupt rqst =
  DATA #1 : Hexdump, 4 bytes
  0x0A0000004DFFD204 : 0000 0000
  ....
  
  FUNCTION: DB2 UDB, buffer dist serv, sqlkdDispatchRequest,
  probe:205
  RETCODE : ZRC=0xFFFFFC48=-952
            SQL0952N  Processing was cancelled due to an
  interrupt.
  
  ...
  FUNCTION: DB2 UDB, SQO Memory Management,
  sqloDiagnoseFreeBlockFailure, probe:10
  MESSAGE : Possible memory corruption detected.
  DATA #1 : ZRC, PD_TYPE_ZRC, 4 bytes
  0x820F0002
  DATA #2 : Corrupt block address, PD_TYPE_CORRUPT_BLK_PTR, 8
  bytes
  0x0a0001000859f1b0
  DATA #3 : Block header, PD_TYPE_BLK_HEADER, 24 bytes
  0x0A0001000859F198 : 0000 FF0C 0000 0000 0A00 0100 0859 F9E8
  .............Y..
  0x0A0001000859F1A8 : 0A00 0100 0859 EE80
  .....Y..
  code: 0xff0c
  curSize: 0
  SMemBlkFileId: 167772416
  SMemBlkLOC: 2137
  SMemBlkNum: 63976
  CodedCSG: a0001000859ee80
  DATA #4 : Data header, PD_TYPE_BLK_DATA_HEAD, 48 bytes
  0x0A0001000859F1B0 : 5441 4F42 0000 0000 0001 F348 0000 0000
  TAOB.......H....
  0x0A0001000859F1C0 : 0000 0000 0000 0000 0000 0000 FF00 0001
  ................
  0x0A0001000859F1D0 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  CALLSTCK: (Static functions may not be resolved correctly, as
  they are resolved to the nearest symbol)
    [0] 0x090000000C685BB4
  @88@sqloDiagnoseFreeBlockFailure__FP8SMemFBlkCb + 0xF8
    [1] 0x090000000E3ACC74 sqlofmblkEx + 0xC
    [2] 0x090000000E323404 sqlridrp__FP8sqlrr_cbP9sqlri_tao +
  0x1B0
    [3] 0x090000000E305EEC sqlridrp__FP8sqlrr_cbP9sqlri_tao + 0x80
    [4] 0x090000000E305C90
  sqlridrp__FP8sqlrr_cbP9sqlri_tao@glue2D4 + 0x3C
    [5] 0x090000000EB9CEB4 sqlricls_complex__FP8sqlrr_cbilN23 +
  0xE24
    [6] 0x090000000F8D85C8
  sqlrr_subagent_router__FP8sqeAgentP12SQLE_DB2RA_T + 0x1B34
    [7] 0x090000000F8D7E30
  sqlrr_subagent_router__FP8sqeAgentP12SQLE_DB2RA_T + 0x139C
    [8] 0x090000000CB95DAC
  @82@sqleSubRequestRouter__FP8sqeAgentPUiPUl + 0x1850
    [9] 0x090000000F4F358C RunEDU__8sqeAgentFv + 0x35A4C
    [10] 0x090000000F473738 RunEDU__8sqeAgentFv + 0x124
    [11] 0x090000000E288AE4 EDUDriver__9sqzEDUObjFv + 0x130
    [12] 0x090000000DF1A2D4 sqloEDUEntry + 0x3A0
    [13] 0x09000000011C8E10 _pthread_body + 0xF0
    [14] 0xFFFFFFFFFFFFFFFC ?unknown + 0xFFFFFFFF
  
  .....
  FUNCTION: DB2 UDB, SQO Memory Management,
  sqloDiagnoseFreeBlockFailure, probe:10
  MESSAGE : ADM14001C  An unexpected and critical error has
  occurred: "Panic".
            The instance may have been shutdown as a result.
  "Automatic" FODC
            (First Occurrence Data Capture) has been invoked and
  diagnostic
            information has been recorded in directory
  
  ....
  

Local fix

• RTC - 561270
  
  Turn of SMP if possible.
  

Problem summary

• Users Affected:
  All
  
  Problem Description:
  Out of Memory in OPS JVM when recoveryBp is run . Continuous
  memory increase due to ActivityData objects growth after each
  Recovery run.
  
  Seeing below error when listing the Active WF object OPS call
  while involving the recovery BP :
  
  WARNING: The OpsServer command timeout has been exceeded for the
  request: GET
  /listwfliveinfo?sig=XAopqUQmiwcVv%2FzUuOcW4ivS%2BawCmPUS8BJnqT81
  RB%2B3RhKNmVZu8LR2ASwD0hZzRdVY1u98bQ4C4isB%2FesFzi92BT4hW3akVgLh
  ZxRCpuctLrFPHIK%2BmER4NfArIf7U67YWuUE8gB4C%2BGzxHNTXRggGv6Cu4hue
  7yaukDSY0ikKt19VRZuWDUozxqabvsmAk%2BUgaLMAVmUwwJM7l4kbkSP3%2FK65
  GMCgIE3OIrfezEeF2N%2FkJjVDO1bpwip%2F74%2FLBEiJoDIaBBHdhoa8Xlm8A1
  7DabSNWzu9ObWbhZUgIsnoPivcyBpAXMfk1s7rUyi2S6rcC9OIYol64KNwec5CRQ
  %3D%3D&seckey=listwfliveinfo%3A1517808634723 HTTP/1.0
  java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at
  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor
  Impl.java:95)
  at
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
  AccessorImpl.java:56)
  at java.lang.reflect.Method.invoke(Method.java:620)
  at
  com.sterlingcommerce.woodstock.ops.server.CommandThread.run(Comm
  andThread.java:58)
  Caused by: java.lang.OutOfMemoryError: Java heap space
  at
  java.lang.StringBuilder.ensureCapacityImpl(StringBuilder.java:34
  2)
  at java.lang.StringBuilder.append(StringBuilder.java:118)
  at
  java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(Objec
  tInputStream.java:3301)
  at
  java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(Objec
  tInputStream.java:3209)
  at
  java.io.ObjectInputStream$BlockDataInputStream.readLongUTF(Objec
  tInputStream.java:3192)
  at
  java.io.ObjectInputStream.readString(ObjectInputStream.java:1763
  )
  at
  java.io.ObjectInputStream.readObject0(ObjectInputStream.java:145
  5)
  at
  java.io.ObjectInputStream.readObject(ObjectInputStream.java:448)
  at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:338)
  at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:187)
  at
  java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod
  (RemoteObjectInvocationHandler.java:240)
  at
  java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjec
  tInvocationHandler.java:192)
  at com.sun.proxy.$Proxy0.getWFThread(Unknown Source)
  at
  com.sterlingcommerce.woodstock.ops.AppServerOps.getWFThread(AppS
  erverOps.java:582)
  ... 5 more
  
  Platforms Affected:
  All
  

Problem conclusion

• Resolution Summary:
  
  A code fix is provided.
  
  
  Delivered In:
  5020603_6
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  IT24955

Fix information

• Fixed component name

  DB2 FOR LUW

• Fixed component ID

  DB2FORLUW

Applicable component levels