Troubleshooting OutOfMemoryError (OOM) that points to com/ibm/rmi/iiop/CDROutputStream Object

Diagnosing The Problem

The com/ibm/rmi/iiop/CDROutputStream IdentityHashtable may have references to some or all of the objects in the graph of the target object. This is a performance feature for value indirection and the number of objects will be based on how much of the target object has been processed and written. This may mean that heap analysis tools will not report those objects in the retained sets of either the CDROutputStream or the target object, but instead in the dominator of those two objects (such as a thread or classloader). To understand the "real size" of the target object in Memory Analyzer Tool:

1. Find the target object by process of elimination or, more deterministically, by taking a system dump and running Java Basics -> Thread Stacks, finding the writeObject method call and taking the object from the frame locals.
2. Create a customized retained set on the target object and exclude (the -x option) the CDROutputStream object.

Similarly, there can be another object graph for CDROutputStream object as below:

<0> [1,576,780,960] 0x7000000358cc8e8 [296] com/ibm/rmi/iiop/CDROutputStream
<1> [571,605,296] 0x7000000358ccb98 [104] com/ibm/rmi/util/buffer/SequentialByteBuffer
<2> [571,605,128] 0x7000000358ccbe8 [64] com/ibm/rmi/util/list/LinkedList
<3> [571,605,064] 0x7000000358cd4b8 [88] com/ibm/rmi/util/buffer/SequentialByteBuffer$Position
<4> [571,604,976] 0x7000000358cd5a8 [88] com/ibm/rmi/util/buffer/SequentialByteBuffer$Position
.........................
<1> [516,120,944] 0x7000000358cd078 [80] com/ibm/rmi/util/IdentityHashtable
<2> [516,120,864] 0x700000061626780 [26,738,712] array of com/ibm/rmi/util/IdentityHashtableEntry
<3> [42,983,200] 0x70000007eef7ca0 [80] com/ibm/rmi/util/IdentityHashtableEntry
<4> [42,982,984] 0x70000007ceac148 [80] com/ibm/rmi/util/IdentityHashtableEntry
.......................
<1> [489,049,784] 0x7000000358cca18 [88] com/ibm/rmi/util/Interop
<2> [489,048,968] 0x700000002d989c0 [720] com/ibm/CORBA/iiop/ORB
<3> [488,954,760] 0x700000002d96e30 [88] java/util/Properties
<4> [488,954,672] 0x700000002d97618 [792] array of java/util/Hashtable$Entry
........................

The large com/ibm/rmi/iiop/CDROutputStream comprises of com/ibm/rmi/util/buffer/SequentialByteBuffer and com/ibm/rmi/util/IdentityHashtable objects

Purpose of SequentialByteBuffer and IdentityHashtable objects:
In case of local invocation, the buffer in use is SequentialByteBuffer. The large sized SequentialByteBuffer is due to large size messages leading to the buffer contents being held up in the memory till the entire message is written out.
Large amount of com/ibm/rmi/util/buffer/SequentialByteBuffer$Position can be a result of large CORBA messages being marshalled/unmarshalled which will be normal. By design, SequentialBytebuffer holds data in "Pieces" of successively double sizes starting with 1KB,2kB,4kb and so on, hence for large-sized messages the latter datapieces are themselves very large in size (LinkedList). This SequentialByeBuffer is eligible for GC once the processing of the request is complete.

IdentityHashtable - valueCache is populated whenever a value is being marshalled and it is held in the memory till the entire message is written out (to support value indirection). Value indirection is used to avoid writing redundant data into the message and hence requires the message to remain in the memory till the entire message is written out. The entries in the IdentityHashtable depends on the size of the message being sent which cannot be controlled/avoided because of the logic used in value indirection.