PH14613: PH14613: Intelligent Management enabled Plugin crashes in multicell environment

9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3

APAR status

• Closed as program error.

Error description

• This issue appears to occur only in a multicell environment
  when load is placed on IHS during a recycle of some set of
  app servers in one of the cells.  Odrlib attempts to rebuild
  the tree leading to a crash of odrlib in the
  odrTargetSelect.tsRemoveVirtualClusterFromMultiVCs() method.
   Here are two relevant stacks from this failure:
  
  #0 0x00007f2f504cdd4a in odrListGetHead (list=0x0) at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL
  ist.c:237
  #1 0x00007f2f504cdece in odrListFind (list=0x0,
  obj=0x7f2f4294f070)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL
  ist.c:366
  #2 0x00007f2f504ce0fa in odrListFindAndRemove (list=0x0,
  obj=0x7f2f4294f070)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL
  ist.c:479
  #3 0x00007f2f504dd9ba in tsRemoveVirtualClusterFromMultiVCs
  (ts=0x7f2f40001940, vc=0x7f2f4294f070)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
  argetSelector.c:2774
  #4 0x00007f2f50499453 in vcAddServerApp (vc=0x7f2f4294f070,
  sapp=0x7f2f4068e8e0)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/Virt
  ualCluster.c:267
  #5 0x00007f2f504d986a in handleLinkServerToWebModuleRequest
  (server=0x7f2f41cb5d80, req=0x7f2f4036c670)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
  argetSelector.c:1345
  #6 0x00007f2f504d99b5 in tsLinkServerToWebModule
  (server=0x7f2f41cb5d80, wm=0x7f2f4294f7d0, add=1 '\001')
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
  argetSelector.c:1372
  #7 0x00007f2f504a6f4c in odrLinkServerToWebModule
  (server=0x7f2f41cb5d80, owm=0x7f2f429504e0, add=1 '\001')
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
  c:2596
  #8 0x00007f2f504a8ce7 in jsonHandleRelationships
  (json=0x7f2f40b06280, pi=0x7f2f4e8e2ba0)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
  c:3358
  #9 0x00007f2f504a8fdd in jsonHandleOp (json=0x7f2f40b2c570,
  pi=0x7f2f4e8e2ba0)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
  c:3410
  #10 0x00007f2f504a91f1 in jsonHandle (json=0x7f2f403929b0,
  pi=0x7f2f4e8e2ba0)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
  c:3446
  #11 0x00007f2f504a9dbd in odrHandleJson
  (json=0x7f2f403929b0,
  jsonStr=0x7f2f401fc190 "{\n  \"add\": {\n
  \"relationships\": {\n     \"server:webModule\": [\n", ' '
  <repeats 12 times>, "{\n", ' ' <repeats 15 times>,
  "\"server\":
  \"/cell/WDE/node/WDEFNCC001/server/WDEFNCC001C1\",\n", ' '
  <repeats 15 times>, "\"webModule\": \"/cell/WDE/app"...,
  c=0x5587fa015ae0)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
  c:3676
  #12 0x00007f2f504ac01c in odrClientProcessHandleMessage
  (odr=0x5587f9fb78a0, msg=0x7f2f4e8e2d30)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
  c:4591
  #13 0x00007f2f504cfb94 in odrClientProcessRun
  (parm=0x5587fa013660)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrM
  ultiProcess.c:422
  #14 0x00007f2f504e0a9a in odrStartRoutineWrapper
  (arg=0x5587fa013570)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
  hread.c:60
  #15 0x00007f2f5fb8ddd5 in start_thread () from
  /lib64/libpthread.so.0
  #16 0x00007f2f5f6b2ead in clone () from /lib64/libc.so.6
  
  and
  
  #0 0x00007f7e4331bc76 in odrHashCompute
  (key=0x44572f6c6c65632f <Address 0x44572f6c6c65632f out of
  bounds>)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrH
  ash.c:287
  #1 0x00007f7e4331bccb in odrHashGet (hm=0x7f7e34001670,
  key=0x44572f6c6c65632f <Address 0x44572f6c6c65632f out of
  bounds>)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrH
  ash.c:300
  #2 0x00007f7e4333e840 in tsGetListOfVcForMultiVc
  (ts=0x7f7e34000fa0, multiVc=0x7f7e34aea2b0)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
  argetSelector.c:2744
  #3 0x00007f7e4333e9a9 in tsRemoveVirtualClusterFromMultiVCs
  (ts=0x7f7e34000fa0, vc=0x7f7e342de040)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
  argetSelector.c:2772
  #4 0x00007f7e432fa453 in vcAddServerApp (vc=0x7f7e342de040,
  sapp=0x7f7e34010ac0)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/Virt
  ualCluster.c:267
  #5 0x00007f7e4333a86a in handleLinkServerToWebModuleRequest
  (server=0x7f7e34292f50, req=0x7f7e34011310)
  at
  /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
  argetSelector.c:1345 (edited)
  ...
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server WAS ND edition- Virtual              *
  *                  Enterprise/IM Component                     *
  ****************************************************************
  * PROBLEM DESCRIPTION: In a multi-cell environment, after one  *
  *                      of the cells is restarted, ODRLIB       *
  *                      component inside WAS IHS intermittently *
  *                      crashes.                                *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  In the ODRLIB component of IHS there is code that is used to
  select a target for incoming requests using stored information
  about the topology of the cells for which it is sending the
  requests.  The software that manages that information had a
  defect when a cell was removed and then added back which led to
  this problem.
  

Problem conclusion

• Changes were made to code that manages the target selection
  endpoints so that when a cell is removed and then added back,
  the correct endpoint object is selected.
  
  The fix for this APAR is currently targeted for inclusion in fix
  pack 9.0.5.1.  Please refer to the Recommended Updates page for
  delivery information:
  https://www-
  01.ibm.com/support/docview.wss?rs=180&uid=swg1PH14613
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels

• R900 PSY

     UP