PK87562: Server appears hung and is not able to process any requests. Controller region has c9c213aa minor code.

6.1.0.29: Java SDK 1.5 SR11 Cumulative Fix for WebSphere Application Server
6.1.0.31: Java SDK 1.5 SR11 FP1 Cumulative Fix for WebSphere Application Server
6.1.0.33: Java SDK 1.5 SR12 FP1 Cumulative Fix for WebSphere
6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• Controller region stops processing requests after c9c213aa minor
  code. The problem is very intermittent and only present during a
  small timing window.  Very likely, once server is recycled,
  the issue will not reoccur.
  
  Trace: 2009/05/22 03:11:57.278 01 t=9BB730 c=UNK key=S2
  (00000004)
     Description: Throw CORBA system exception
     exception id: CORBA::INTERNAL
     minor code:
     from filename: ./bboobokc.cpp
     at line: 1075
   Trace: 2009/05/22 03:11:57.305 01 t=9BB730 c=UNK key=S2
  (04072019)
     Description: Java exception created
     from filename: ./bboocjex.cpp
     Method Name: createExceptionForJava(JNIEnv *, char *,
  unsigned int,
  CORBA::CompletionStatus, CORBA::Exception *, char *, CORBA::Bo
   olean, CORBA::Boolean)
   org.omg.CORBA.INTERNAL: Exception occured during native method
  call from JAVA  vmcid: 0xc9c21000  minor code: 3AA  completed:
  No
    at com.ibm.ws390.orb.BOSSObjectKey.buildBOSSObjectKey
    at com.ibm.ws390.orb.BOSSObjectKey.buildBOSSObjectKey
    at com.ibm.ws390.orb.BOSSObjectKey.length(Unknown Source)
    at com.ibm.rmi.Profile.write(Profile.java:669)
    at com.ibm.rmi.IOR.write(IOR.java:416)
    at com.ibm.ws390.orb.ClientDelegate.createRequest
    at com.ibm.ws390.orb.ClientDelegate.request(Unknown Source)
    at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java
       :458)
    at com.ibm.ws390.management.connector.corba._CorbaConnectorStu
       b.invoke(_CorbaConnectorStub.java:619)
    at com.ibm.ws390.management.ServantMBeanInvoker.invokeServantM
       Bean(ServantMBeanInvoker.java:1085)
    at com.ibm.ws390.management.ServantMBeanInvoker.invokeSpecifie
       dServants(ServantMBeanInvoker.java:739)
    at com.ibm.ws.management.component.JVMMBeanProxy.isStateManage
       able(JVMMBeanProxy.java:616)
  
  This c9c213aa minor code may be preceded by these messages,
  though on their own, these are not an error and are not being
  addressed by this apar:
  
  Trace: 2009/05/22 03:08:55.504 01 t=9BBA60 c=UNK key=S2
  (13007002)
     ThreadId: 0000003a
     FunctionName:
  com.ibm.ws.channel.framework.impl.WSChannelFrameworkImpl
     SourceId:
  com.ibm.ws.channel.framework.impl.WSChannelFrameworkImpl
     Category: AUDIT
     ExtendedMessage: BBOO0222I: CHFW0019I: The Transport Channel
  Service has started chain IIOP_OUT_df9c4327.
   Trace: 2009/05/22 03:08:55.593 01 t=9BB8C8 c=UNK key=S2
  (13007002)
     ThreadId: 0000003b
     FunctionName: com.ibm.ws390.ziop.ZIOPChainManager
     SourceId: com.ibm.ws390.ziop.ZIOPChainManager
     Category: SEVERE
     ExtendedMessage: BBOO0220E: CWZGB0001E: The IIOP channel
  transport has not been configured correctly: Channel already
  exists: IIOP_OUT_df9c4327-ziop
  
  This issue is not present in V7
  

Local fix

• Try to restart the server as this issue is caused by a small
  timing window, so likely next restart will be ok.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server V6.1.0                               *
  ****************************************************************
  * PROBLEM DESCRIPTION: Controller suffers CORBA::INTERNAL      *
  *                      C9C213AA.                               *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The WebSphere Application Server for z/OS controller address
  space suffers a CORBA::INTERNAL exception with minor code
  C9C213AA.
  
  The exception's Java stack traceback may resemble this:
  
  org.omg.CORBA.INTERNAL: Exception occured during native method
  call from JAVA
  vmcid: 0xc9c21000  minor code: 3AA  completed: No
  at com.ibm.ws390.orb.BOSSObjectKey.buildBOSSObjectKey
  at com.ibm.ws390.orb.BOSSObjectKey.buildBOSSObjectKey
  at com.ibm.ws390.orb.BOSSObjectKey.length
  at com.ibm.rmi.Profile.write
  at com.ibm.rmi.IOR.write
  at com.ibm.ws390.orb.ClientDelegate.createRequest
  at com.ibm.ws390.orb.ClientDelegate.request
  at org.omg.CORBA.portable.ObjectImpl._request
  at com.ibm.ws390.management.connector.corba._CorbaConnector...
  at com.ibm.ws390.management.ServantMBeanInvoker.invokeServa...
  at com.ibm.ws390.management.ServantMBeanInvoker.invokeSpeci...
  at com.ibm.ws.management.component.JVMMBeanProxy.isStateMan...
  
  Some controller facilities may become unresponsive after the
  exception.
  
  The INTERNAL exception is caused by a timing window in the
  object key creation code.  Two threads building the same
  object key using the same storage buffer may conflict with
  each other, causing the C9C213AA INTERNAL exception.
  

Problem conclusion

• The timing window in the object key creation code was removed.
  
  APAR PK87562 is currently targeted for inclusion in
  Service Level (Fix Pack) 6.1.0.28 of WebSphere
  Application Server V6.1
  
  Please refer to URL:
  //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
  for Fix Pack availability.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UK50733

Fix information

• Fixed component name

  WEBSPHERE FOR Z

• Fixed component ID

  5655I3500

Applicable component levels

• R610 PSY UK50733

     UP09/10/27 P F910

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.