Important facts about garbage collection in WebSphere Message Broker
vivekgrover 120000FG70 Comments (17) Visits (25336)
In WebSphere Message Broker, when an execution group is started it creates a JVM which is primarily used by the IBM primitive nodes that make use of java functionality. Examples of these include:
DataFlowEngine JVM can be configured by either passing parameters to the DataFlowEngine directly or through the broker. When using the broker JVM by any of the means above, DataFlowEngine memory may continue to grow and very often may cause resource problems.
Garbage collection is a process of automatic memory management within JVM. The garbage collector attempts to reclaim the memory used by the objects that would never be accessed again by the broker. Any objects that are created during the execution of java functionality in Message Broker will be created within its JVM Heap. Even if no java based functionality has been implemented in a message flow, some of the internal operations in the Execution group use java. The administration threads that perform these internal operations run periodically and when they run they also make allocations on the JVM heap. When the garbage collection is driven, the java application frees up all of its resources, which would then be flagged for deletion. The DataFlowEngine JVM does not delete these resources from the JVM heap until garbage collection is driven. This means, when the garbage collection is driven, it will free up the blocks on the DataFlowEngine JVM heap. Until then, these blocks cannot be re-used, and so the JVM will continue to grow towards its maxJVMHeapSize.
How does garbage collection process work?
The JVM Heap size must be large enough so that all of the threads can perform their java operations successfully when processing concurrently, but must not be so large that the JVM Heap consumes most of the address space. For example, it is possible to set the maxJvmHeapSize of a 32-bit DataFlowEngine to 2GB, but a 32-bit DataFlowEngine process can only grow up to 2GB. So with this type of setting, the JVM can consume most of the address space and hence leave the native C++ memory heap with very little remaining space. This could also cause the DataFlowEngine to abend reporting 'Failed to allocate memory'. So, if the maxJvmHeapSize is too small then java OutOfMemory exceptions will be thrown, however if it is set to a large value, then this can also lead to abends reporting 'Failed to allocate memory'.
To see how often the JVM is doing garbage collection, you can activate the resource statistics collection and check the Cumu
To see the current garbage collection, you can monitor the Cumu
If the UsedMemoryInMB value is always less than the InitialMemoryInMB value, then it would mean that you may have allocated more memory for the heap than required. Therefore, you can reduce the jvmMinHeapSize value for the execution group to a value that is closer to the UsedMemoryInMB value using mqsi
Using clearMessage() in custom Java programs
If your DataFlowEngine process continually grows until an OutOfMemory error is reached, then it is likely that the JVM heap has been exhausted. This could mean that the JVM garbage collection was not driven, indicating the objects are not being freed within the plugin or java Compute node.
When you write your own java code, then care must be taken to ensure all the variables are dereferenced so that the storage can be freed by the garbage collector. Some java objects may require methods to be called on them to delete aspects of the storage created for the object. Sometimes, you may also want to free up the message objects sooner rather than waiting for the garbage collector to be activated. For this purpose, clearMessage() method can be used. If the message object is not deleted using clearMessage() function, then it will not be freed until the Garbage Collector is called. An example of this in the broker java classes is the MbMessage object. This needs to have a clearMessage() call on it so that the associated C++ objects and message bitstream are deleted. For example, if your java node (plugin/JCN) is creating a new output message then you would have lines like:
MbMessage newMsg = null;
In this case, if the createMessage() method is called, a buffer would have been allocated ready to serialize the message tree. As indicated earlier, for every createMessage() issued, an associated clearMessage() needs to be called. This needs to be done after the "outTerm.propagate" and regardless of whether an exception is thrown or not. For example:
MbMessage newMsg = null;
Capturing garbage collection activity
This will log the garbage collection activity in the file gc.trc.
You can also capture the GC activity for all the Execution Groups together, such as at the broker level. The GC activity at the broker level can be captured in one of the following ways:
Understanding garbage collection data
<AF: Allocation Failure. need 16400 bytes, 1420700 ms since last AF>
You can also use the IBM Support Assistant (ISA) workbench, a GUI based application to view the GC activity in graphical form. The ISA workbench can be downloaded from the following IBM site:
After download, you can go to Update > Find updates and add on to install "IBM Monitoring and Diagnostic Tools for Java - Garbage collection and Memory Visualizer". This will install the tool that can now be used to view the garbage collection activity. You can use this tool to analyze any stdout, console.txt or gc output file captured as described earlier. To see the screen-capture that shows the graph created by the tool with the actual heap set on the broker and the used heap during a message flow run, click here. You can also use the Resource statistics as explained above to view the graphs and analyze the JVM usage activity.
I hope this information is helpful in analyzing java memory problems. Let me know if you have any questions.