IT28543: Start HADR hang after upgrade to v11.1.4.4

Db2 Version 11.1 Mod 4 Fix Pack 5 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 6 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 6 for Linux, UNIX, and Windows

APAR status

• Closed as program error.

Error description

• When using Db2 v11.1.4.4 or above, this issue may be
  encountered. HADR start can hang on the primary or standby.
  
  The following stack message is generated:
  
  <StackTrace>
  -----FUNC-ADDR---- ------FUNCTION + OFFSET------
  0x00007FBB6555ACA6
  _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
  0x0356
  		(/db2/omddb01b/home/sqllib/lib64/libdb2osse.so.1)
  0x00007FBB6555A8EB ossDumpStackTraceV98 + 0x002b
  		(/db2/omddb01b/home/sqllib/lib64/libdb2osse.so.1)
  0x00007FBB65556AD7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
  0x00d7
  		(/db2/omddb01b/home/sqllib/lib64/libdb2osse.so.1)
  0x00007FBB6CDBE187 sqlo_trce + 0x03c7
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6CF29C0B sqloDumpDiagInfoHandler + 0x020b
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB74ABF5D0 address: 0x00007FBB74ABF5D0 ; dladdress:
  0x00007FBB74AB0000 ; offset in lib: 0x000000000000F5D0 ;
  		(/lib64/libpthread.so.0)
  0x00007FBB6CE2DD42 sqloPdbTcpIpGetAddrInfo + 0x0082
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6CE2D0CB sqloPdbInitNodeAddrHndl + 0x01ab
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB67144519
  _Z25hdrResolveHostNameAndPort19SQLOPDBNODEADDRTYPEPcS0_P18HDR_CO
  MM_FULL_ADDRPS0_ + 0x0159
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6711F153
  _Z23hdrResolveHostNamesToIp19SQLOPDBNODEADDRTYPEPcS0_S0_S0_P17HD
  R_HOST_SVC_PAIRmP24HDR_ALL_RESOLVED_IP_PORTP21hdrHostNameResolve
  Rcs + 0x0073
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6711EC28
  _Z34hdrResolveAllHostsAndSetIpProtocolP11HADR_SHAREDjPcS1_S1_S1_
  P17HDR_HOST_SVC_PAIRmP24HDR_ALL_RESOLVED_IP_PORT + 0x00b8
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6711E70A
  _Z36hdrValidateConfiguredRemoteAddressesP16sqeLocalDatabaseP14sq
  lpMasterDbcbj16hdrDbCfgValidatePKcP24HDR_ALL_RESOLVED_IP_PORTRm
  + 0x042a
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6711CCCC
  _Z13hdrEduStartupP16sqeLocalDatabaseP9sqeBsuEduP14sqlpMasterDbcb
  bjb19hdrEduStartupActionP5sqlcab + 0x05dc
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6D167E94
  _Z20sqlpCheckToStartHadrP8sqeAgentP16sqeLocalDatabaseP5sqlcabjPb
  b + 0x0724
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B1FDD80
  _Z8sqledintP8sqeAgentP16sqeLocalDatabaseP5sqlcacPciPbm + 0x2360
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B1F392D
  _ZN16sqeLocalDatabase12FirstConnectEP8SQLE_BWARcP8sqeAgentP8sqlo
  _gmtiiPb + 0x52ad
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B1D44DD
  _ZN8sqeDBMgr23StartUsingLocalDatabaseEP8SQLE_BWAP8sqeAgentRccP8s
  qlo_gmtPb + 0x0f5d
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B0FBECE
  _ZN14sqeApplication13AppStartUsingEP8SQLE_BWAP8sqeAgentccP5sqlca
  Pc + 0x044e
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B10F02C
  _Z22sqleProcessConnectTypeP14db2UCinterfaceP8SQLE_BWAP8sqeAgentP
  5sqlcaP22SQLELOST_STATUS_VECTOR + 0x00ac
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B0FA560
  _ZN14sqeApplication13AppLocalStartEP14db2UCinterface + 0x07b0
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B497D8F _Z11sqlelostWrpP14db2UCinterface + 0x004f
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B49DC41 _Z14sqleUCengnInitP14db2UCinterfacet + 0x0891
  
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B49429C sqleUCagentConnect + 0x089c
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B738EDD
  _Z18sqljsConnectAttachP13sqljsDrdaAsCbP14db2UCinterface + 0x02ed
  
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B6C8A6B
  _Z16sqljs_ddm_accsecP14db2UCinterfaceP13sqljDDMObject + 0x008b
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B6BCC2D
  _Z17sqljsParseConnectP13sqljsDrdaAsCbP13sqljDDMObjectP14db2UCint
  erface + 0x01ed
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B6BDE4E
  _Z10sqljsParseP13sqljsDrdaAsCbP14db2UCinterfaceP8sqeAgentb +
  0x054e
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B6B0D2D address: 0x00007FBB6B6B0D2D ; dladdress:
  0x00007FBB65AB4000 ; offset in lib: 0x0000000005BFCD2D ;
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B6B7313 address: 0x00007FBB6B6B7313 ; dladdress:
  0x00007FBB65AB4000 ; offset in lib: 0x0000000005C03313 ;
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B6B7FCF _Z17sqljsDrdaAsDriverP18SQLCC_INITSTRUCT_T +
  0x011f
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6B0E19E3 _ZN8sqeAgent6RunEDUEv + 0x0de3
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6EA0121E _ZN9sqzEDUObj9EDUDriverEv + 0x01be
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB6CF276BA sqloEDUEntry + 0x057a
  		(/db2/omddb01b/home/sqllib/lib64/libdb2e.so.1)
  0x00007FBB74AB7DD5 address: 0x00007FBB74AB7DD5 ; dladdress:
  0x00007FBB74AB0000 ; offset in lib: 0x0000000000007DD5 ;
  		(/lib64/libpthread.so.0)
  0x00007FBB64562EAD clone + 0x006d
  		(/lib64/libc.so.6)
  </StackTrace>
  

Local fix

• Using an IP address instead of a hostname string should avoid
  this issue. Change the following to use an IP address:
  HADR_LOCAL_HOST / HADR_REMOTE_HOST / HADR_TARGET_LIST
  That being said, there is some chance that this workaround would
  not be suitable for virtual IP scenarios, since there will still
  be a need for resolving the real IP address.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * ALL                                                          *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade to Db2 11.1 Mod 4 Fixpack 5 or higher                *
  ****************************************************************
  

Problem conclusion

• First fixed in Db2 11.1 Mod 4 Fixpack 5
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  DB2 FOR LUW

• Fixed component ID

  DB2FORLUW

Applicable component levels

• RB10 PSN

     UP