IV65773: EEZJ0604W AND EEZF0006E A JAVA.RMI.REMOTEEXCEPTION: CORBA NO_MEMORY

SAAM4106_Verification_Signatures
4.1.0-TIV-SAAMA-AIX-FP0006
4.1.0-TIV-SAAMR-Windows-FP0006
4.1.0-TIV-SAAMR-AIX-FP0006
4.1.0-TIV-SAAMR-LinuxS390X-FP0006
4.1.0-TIV-SAAMR-LinuxPPC64LE-FP0006
4.1.0-TIV-SAAMR-LinuxPPC64-FP0006
4.1.0-TIV-SAAM-LinuxS390X-FP0006
4.1.0-TIV-SAAMR-LinuxX86_64-FP0006
4.1.0-TIV-SAAM-AIX-FP0006
4.1.0-TIV-SAAM-LinuxX86_64-FP0006
4.1.0-TIV-SAAMA-AIX-FP0003
4.1.0-TIV-SAAMR-LinuxS390X-FP0003
4.1.0-TIV-SAAMR-LinuxX86_64-FP0003
4.1.0-TIV-SAAMR-LinuxPPC64-FP0003
4.1.0-TIV-SAAMR-Windows-FP0003
4.1.0-TIV-SAAMR-AIX-FP0003
4.1.0-TIV-SAAM-LinuxS390X-FP0003
4.1.0-TIV-SAAM-LinuxX86_64-FP0003
4.1.0-TIV-SAAM-AIX-FP0003
4.1.0-TIV-SAAM-AIX-FP0002
4.1.0-TIV-SAAMA-AIX-FP0002
4.1.0-TIV-SAAMR-Windows-FP0002
4.1.0-TIV-SAAMR-LinuxS390-FP0002
4.1.0-TIV-SAAMR-LinuxPPC-FP0002
4.1.0-TIV-SAAMR-LinuxI386-FP0002
4.1.0-TIV-SAAMR-AIX-FP0002
4.1.0-TIV-SAAM-LinuxI386-FP0002
4.1.0-TIV-SAAM-LinuxS390-FP0002
4.1.0-TIV-SAAM-AIX-FP0002.fo.xml
4.1.0-TIV-SAAMR-Windows-FP0001
4.1.0-TIV-SAAMR-LinuxS390-FP0001
4.1.0-TIV-SAAMR-LinuxPPC-FP0001
4.1.0-TIV-SAAMR-LinuxI386-FP0001
4.1.0-TIV-SAAMR-AIX-FP0001
4.1.0-TIV-SAAMA-Windows-FP0001
4.1.0-TIV-SAAMA-Solaris-FP0001
4.1.0-TIV-SAAMA-AIX-FP0001
4.1.0-TIV-SAAM-LinuxS390-FP0001
4.1.0-TIV-SAAM-LinuxI386-FP0001
4.1.0-TIV-SAAM-AIX-FP0001
4.1.0-TIV-SAAM-FP0001, Tivoli System Automation Application Manager

APAR status

• Closed as program error.

Error description

• an endless loop leads to hanging threads and process crash due
  to Out Of Memory situation.
  The SystemOut.log shows
   EEZMDBMsg  E  EEZEventHandlerBean processResAddedEvent
     EEZJ0107E Forwarding an event to the end-to-end automation
     domain TSAAM_Test failed.
     Details about the exception:
      com.ibm.eez.sdk.exceptions.EEZConnectionFailedException:
      EEZF0006E A java.rmi.RemoteException: CORBA NO_MEMORY
      0x4942f982 No; nested exception is:
       org.omg.CORBA.NO_MEMORY:
       >> SERVER (id=12345678, host=xx.xx.xx.xx) TRACE START:
       >>   org.omg.CORBA.NO_MEMORY: Unable to read value from
          underlying bridge : Java heap space  vmcid: IBM
       minor code: 982 completed: No
     >>  at com.ibm.rmi.iiop.CDRReader.read_value
        (CDRReader.java:2429)
     >>  at com.ibm.eez.dmn.client._EEZDaemon_Tie.handleEvent
        (_EEZDaemon_Tie.java:587)
    ...
   Threads hang:
   EEZAsyncMsg   W  EEZThreadMessenger logThreadNumMessage
    EEZJ0604W There are 4 active threads that are managed by
    component EEZWatchdogSessionBean and may be hung.
  
   ... number increases
  
   EEZAsyncMsg   W  EEZThreadMessenger logThreadNumMessage
    EEZJ0604W There are 369 active threads that are managed by
    component EEZWatchdogSessionBean and may be hung.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: Tivoli System Automation Application Manager
  * (SA AM) 4.1 users
  ****************************************************************
  * PROBLEM DESCRIPTION:
  * The SA AM automation engine might end up in an infinite loop,
  * making the communication to the first level automation domains
  * impossible. The WebSphere Application server 'SystemOut.log'
  * shows an EEZConnectionFailedException:
  *   com.ibm.eez.sdk.exceptions.EEZConnectionFailedException:
  *   EEZF0006E A java.rmi.RemoteException: CORBA NO_MEMORY
  *   0x4942f982 No; nested exception is:
  *    org.omg.CORBA.NO_MEMORY:
  *    >> SERVER (id=12345678, host=xx.xx.xx.xx) TRACE START:
  *    >>   org.omg.CORBA.NO_MEMORY: Unable to read value from
  *     underlying bridge : Java heap space  vmcid: IBM
  *     minor code: 982 completed: No
  *    >>  at com.ibm.rmi.iiop.CDRReader.read_value
  *     (CDRReader.java:2429)
  *    >>  at com.ibm.eez.dmn.client._EEZDaemon_Tie.handleEvent
  *     (_EEZDaemon_Tie.java:587)
  *
  * The SA AM traces do show the message with ID CHRM1058I, and a
  * resource reference which has its ObservedState switching
  * between 'WasOnline' and 'Online'.
  ****************************************************************
  * RECOMMENDATION:
  ****************************************************************
  

Problem conclusion

• The internal automation Engine (Logic deck) has been adjusted to
  address this issue. The trigger to enter the infinite loop has
  been removed.
  .
  The official fix for this problem is included in Tivoli System
  Automation Application Manager 4.1 fix pack 1.
  | 4.1.0-TIV-SAAM-FP0001 |
  .
  Additional Search Keywords
  .
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SYS AUTO APPLI

• Fixed component ID

  5724S9200

Applicable component levels