Topic
  • 2 replies
  • Latest Post - ‏2012-05-25T16:08:50Z by SystemAdmin
SystemAdmin
SystemAdmin
102 Posts

Pinned topic APC in a multi-cell performance management configuration

‏2012-05-24T20:49:16Z |
In a star topology with multi-cell performance management configured, we are having a problem with dynamic cluster members being started. We have configured the dynamic cluster with lazy start, accessed the application, and no run time task is generated (cluster is in supervised mode).

All other functions/features of VE seem to be working, but not application placement.

We've bounced the JVM running the APC in the center cell, sync'd the nodes, etc. Nothing is working.

We did notice that the APCs in the point cells have values of '-1' in the Runtime tab. We assume this is a fail safe for disabling them even if they are marked as enabled.

Besides the SystemOut.log of the JVM where the APC is running, what else should we look at as far as logs, traces, etc.?

The following error message is displayed in the SystemOut.log file where the APC is running in the center cell when we attempt to hit the application via browser:

APCXDPlaceMap E DCPC0006E: The following error occurred during an attempt to complete the control loop: 258 : unknown.

There is an ODR routing policy in front of this that is routing to a given cell based on the header$host matching a pattern.
Updated on 2012-05-25T16:08:50Z at 2012-05-25T16:08:50Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    102 Posts

    Re: APC in a multi-cell performance management configuration

    ‏2012-05-25T14:15:05Z  
    Here is the FFDC that is generated just before the above first error message in the dmgr SystemOut.log of the center cell hosting the ODRs:

    5/24/12 21:04:28:502 EDT FFDC Exception:com.ibm.apc.helpers.exceptions.APCExceptionSourceId:com.ibm.apc.controller.APCCore.runOneCycle ProbeId:110
    com.ibm.apc.helpers.exceptions.APCException: com.ibm.apc.helpers.exceptions.APCException: java.lang.NullPointerException
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.createSuperMap(APCPlaceMapBuilder.java:303)
    at com.ibm.apc.solver.abs.APCOneShotSolverAbstract.prepareYourself(APCOneShotSolverAbstract.java:75)
    at com.ibm.apc.solver.oneshot.APCOneShotSolverImpl.fire(APCOneShotSolverImpl.java:111)
    at com.ibm.apc.controller.APCCore.runOneCycle(APCCore.java:242)
    at com.ibm.apc.controller.APCRunnable.run(APCRunnable.java:239)
    at java.lang.Thread.run(Thread.java:736)
    Caused by: com.ibm.apc.helpers.exceptions.APCException: java.lang.NullPointerException
    at com.ibm.apc.xd.mapbuild.APCXDPlaceMapSubBuilder.goBuild(APCXDPlaceMapSubBuilder.java:269)
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.runOneSubBuilder(APCPlaceMapBuilder.java:367)
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.runAllSubBuilders(APCPlaceMapBuilder.java:363)
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.createSuperMap(APCPlaceMapBuilder.java:295)
    ... 5 more
    Caused by: java.lang.NullPointerException
    at com.ibm.apc.xd.mapbuild.APCXDPlaceMapSubBuilder.doPlaceMapTransactionalCntrsAndApplsForClusters(APCXDPlaceMapSubBuilder.java:1112)
    at com.ibm.apc.xd.mapbuild.APCXDPlaceMapSubBuilder.goBuild(APCXDPlaceMapSubBuilder.java:256)
    ... 8 more

    Reporter:null

    So we're wondering if the null pointer exception is having a bearing on the ability of the APC to determine when dynamic cluster members should be started.
  • SystemAdmin
    SystemAdmin
    102 Posts

    Re: APC in a multi-cell performance management configuration

    ‏2012-05-25T16:08:50Z  
    Here is the FFDC that is generated just before the above first error message in the dmgr SystemOut.log of the center cell hosting the ODRs:

    5/24/12 21:04:28:502 EDT FFDC Exception:com.ibm.apc.helpers.exceptions.APCExceptionSourceId:com.ibm.apc.controller.APCCore.runOneCycle ProbeId:110
    com.ibm.apc.helpers.exceptions.APCException: com.ibm.apc.helpers.exceptions.APCException: java.lang.NullPointerException
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.createSuperMap(APCPlaceMapBuilder.java:303)
    at com.ibm.apc.solver.abs.APCOneShotSolverAbstract.prepareYourself(APCOneShotSolverAbstract.java:75)
    at com.ibm.apc.solver.oneshot.APCOneShotSolverImpl.fire(APCOneShotSolverImpl.java:111)
    at com.ibm.apc.controller.APCCore.runOneCycle(APCCore.java:242)
    at com.ibm.apc.controller.APCRunnable.run(APCRunnable.java:239)
    at java.lang.Thread.run(Thread.java:736)
    Caused by: com.ibm.apc.helpers.exceptions.APCException: java.lang.NullPointerException
    at com.ibm.apc.xd.mapbuild.APCXDPlaceMapSubBuilder.goBuild(APCXDPlaceMapSubBuilder.java:269)
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.runOneSubBuilder(APCPlaceMapBuilder.java:367)
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.runAllSubBuilders(APCPlaceMapBuilder.java:363)
    at com.ibm.apc.controller.mapbuild.APCPlaceMapBuilder.createSuperMap(APCPlaceMapBuilder.java:295)
    ... 5 more
    Caused by: java.lang.NullPointerException
    at com.ibm.apc.xd.mapbuild.APCXDPlaceMapSubBuilder.doPlaceMapTransactionalCntrsAndApplsForClusters(APCXDPlaceMapSubBuilder.java:1112)
    at com.ibm.apc.xd.mapbuild.APCXDPlaceMapSubBuilder.goBuild(APCXDPlaceMapSubBuilder.java:256)
    ... 8 more

    Reporter:null

    So we're wondering if the null pointer exception is having a bearing on the ability of the APC to determine when dynamic cluster members should be started.
    It appears that the APC stopped working properly after applying VE Fix Pack 2 for v7.0.

    The fix pack is going to be backed out to see if that fixes the issue.