IBM Support

PH14613: PH14613: Intelligent Management enabled Plugin crashes in multicell environment

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • This issue appears to occur only in a multicell environment
    when load is placed on IHS during a recycle of some set of
    app servers in one of the cells.  Odrlib attempts to rebuild
    the tree leading to a crash of odrlib in the
    odrTargetSelect.tsRemoveVirtualClusterFromMultiVCs() method.
     Here are two relevant stacks from this failure:
    
    #0 0x00007f2f504cdd4a in odrListGetHead (list=0x0) at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL
    ist.c:237
    #1 0x00007f2f504cdece in odrListFind (list=0x0,
    obj=0x7f2f4294f070)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL
    ist.c:366
    #2 0x00007f2f504ce0fa in odrListFindAndRemove (list=0x0,
    obj=0x7f2f4294f070)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrL
    ist.c:479
    #3 0x00007f2f504dd9ba in tsRemoveVirtualClusterFromMultiVCs
    (ts=0x7f2f40001940, vc=0x7f2f4294f070)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
    argetSelector.c:2774
    #4 0x00007f2f50499453 in vcAddServerApp (vc=0x7f2f4294f070,
    sapp=0x7f2f4068e8e0)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/Virt
    ualCluster.c:267
    #5 0x00007f2f504d986a in handleLinkServerToWebModuleRequest
    (server=0x7f2f41cb5d80, req=0x7f2f4036c670)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
    argetSelector.c:1345
    #6 0x00007f2f504d99b5 in tsLinkServerToWebModule
    (server=0x7f2f41cb5d80, wm=0x7f2f4294f7d0, add=1 '\001')
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
    argetSelector.c:1372
    #7 0x00007f2f504a6f4c in odrLinkServerToWebModule
    (server=0x7f2f41cb5d80, owm=0x7f2f429504e0, add=1 '\001')
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
    c:2596
    #8 0x00007f2f504a8ce7 in jsonHandleRelationships
    (json=0x7f2f40b06280, pi=0x7f2f4e8e2ba0)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
    c:3358
    #9 0x00007f2f504a8fdd in jsonHandleOp (json=0x7f2f40b2c570,
    pi=0x7f2f4e8e2ba0)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
    c:3410
    #10 0x00007f2f504a91f1 in jsonHandle (json=0x7f2f403929b0,
    pi=0x7f2f4e8e2ba0)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
    c:3446
    #11 0x00007f2f504a9dbd in odrHandleJson
    (json=0x7f2f403929b0,
    jsonStr=0x7f2f401fc190 "{\n  \"add\": {\n
    \"relationships\": {\n     \"server:webModule\": [\n", ' '
    <repeats 12 times>, "{\n", ' ' <repeats 15 times>,
    "\"server\":
    \"/cell/WDE/node/WDEFNCC001/server/WDEFNCC001C1\",\n", ' '
    <repeats 15 times>, "\"webModule\": \"/cell/WDE/app"...,
    c=0x5587fa015ae0)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
    c:3676
    #12 0x00007f2f504ac01c in odrClientProcessHandleMessage
    (odr=0x5587f9fb78a0, msg=0x7f2f4e8e2d30)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odr.
    c:4591
    #13 0x00007f2f504cfb94 in odrClientProcessRun
    (parm=0x5587fa013660)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrM
    ultiProcess.c:422
    #14 0x00007f2f504e0a9a in odrStartRoutineWrapper
    (arg=0x5587fa013570)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
    hread.c:60
    #15 0x00007f2f5fb8ddd5 in start_thread () from
    /lib64/libpthread.so.0
    #16 0x00007f2f5f6b2ead in clone () from /lib64/libc.so.6
    
    and
    
    #0 0x00007f7e4331bc76 in odrHashCompute
    (key=0x44572f6c6c65632f <Address 0x44572f6c6c65632f out of
    bounds>)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrH
    ash.c:287
    #1 0x00007f7e4331bccb in odrHashGet (hm=0x7f7e34001670,
    key=0x44572f6c6c65632f <Address 0x44572f6c6c65632f out of
    bounds>)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrH
    ash.c:300
    #2 0x00007f7e4333e840 in tsGetListOfVcForMultiVc
    (ts=0x7f7e34000fa0, multiVc=0x7f7e34aea2b0)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
    argetSelector.c:2744
    #3 0x00007f7e4333e9a9 in tsRemoveVirtualClusterFromMultiVCs
    (ts=0x7f7e34000fa0, vc=0x7f7e342de040)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
    argetSelector.c:2772
    #4 0x00007f7e432fa453 in vcAddServerApp (vc=0x7f7e342de040,
    sapp=0x7f7e34010ac0)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/Virt
    ualCluster.c:267
    #5 0x00007f7e4333a86a in handleLinkServerToWebModuleRequest
    (server=0x7f7e34292f50, req=0x7f7e34011310)
    at
    /blddir/WAS90.NATV/NATV/ws/code/plugins.http/odrlib/src/odrT
    argetSelector.c:1345 (edited)
    ...
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server WAS ND edition- Virtual              *
    *                  Enterprise/IM Component                     *
    ****************************************************************
    * PROBLEM DESCRIPTION: In a multi-cell environment, after one  *
    *                      of the cells is restarted, ODRLIB       *
    *                      component inside WAS IHS intermittently *
    *                      crashes.                                *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    In the ODRLIB component of IHS there is code that is used to
    select a target for incoming requests using stored information
    about the topology of the cells for which it is sending the
    requests.  The software that manages that information had a
    defect when a cell was removed and then added back which led to
    this problem.
    

Problem conclusion

  • Changes were made to code that manages the target selection
    endpoints so that when a cell is removed and then added back,
    the correct endpoint object is selected.
    
    The fix for this APAR is currently targeted for inclusion in fix
    pack 9.0.5.1.  Please refer to the Recommended Updates page for
    delivery information:
    https://www-
    01.ibm.com/support/docview.wss?rs=180&uid=swg1PH14613
    

Temporary fix

Comments

APAR Information

  • APAR number

    PH14613

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    900

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2019-07-17

  • Closed date

    2019-10-22

  • Last modified date

    2020-01-21

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R900 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"900","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
15 October 2021