Although this blog is self sufficient and would encourage the readers to know more about using memory analyzer tools particularly with respect to heap dump analysis to detect memory leaks, it would be advisable to read the previous blog in this series titled - "Build Enterprise applications with JDO - best practices to avoid memory leaks" to get a better understanding of the causes due to memory related issues in today's enterprise applications using JDO.
This blog essentially continues from the previous blog mentioned above with the troubleshooting section as below -
Verbose gc analysis:
Java writes all the information pertaining to garbage collection process in a log file called native_stderr.log (hereafter called verbose gc log). This file is placed in the same directory as that of System out logs of the application. It contains information such as the heap occupancy when the allocation failure occurred, time taken to complete the each of the stages of GC cycle such as mark / sweep & compact, free heap memory available after collection and so on.
Analyzing verbose gc logs helps understand the pattern of JVM heap utilization. With this information, one can determine whether a memory leak is occurring or not by observing a continuous growth of heap memory over time until exhaustion.
If this is not a pattern which is seen, and a saw tooth pattern is observed, where memory is consumed and released periodically, then the out of memory error may have occurred due to insufficiency of heap memory. If gencon is the GC algorithm being used, then the size of the nursery / tenure area within the heap needs to be appropriately set, so as to avoid out of memory errors.
Below figure shows a sample application’s – memory utilization pattern, graphically obtained after running a parser tool called Pattern Modeling and Analysis tool for Java Garbage collector (PMAT) on Verbose GC log.
The recommended approach taken while troubleshooting a memory leak is by analyzing heap dumps. Heap dumps are automatically generated when the JVM runs out of heap memory and crashes as a result. They can also be taken manually by using application server administration scripts.
A heap dump file would consist of all the objects that were residing in the heap when the out of memory error occurred. Analyzing a sequence of heap dumps taken after a regular interval will help in understanding the size of objects and the number of objects in memory. This in turn, will help in identifying which class / module is leaking memory.
Analyzing heap dumps may reveal that either the code in the application is leaking memory (mostly due to not closing the resources), or a framework on which the application is built is leaking memory, or maybe even so that there maybe a bug in the application server due to which memory is leaking. Narrowing down to this level and then tracing to the exact class / module where the code leaks memory must be done very carefully with proper understanding of the application and its behavior over the end to end request flow, so as to be able to arrive at the right problem areas.
Below are sample screenshots of heapdump analysis pertaining to JDO based application memory leak. This data is obtained by using a tool on heapdump files which parses the entire file to search for leak suspects and displays it in good readable format for ease of analysis and problem identification purposes. This tool is called – Memory Analyzer tool (MAT)
Using memory analyzer tool (MAT), it is very easy to understand the objects that are lying the heap and their memory sizes at that given point in time when the heap dump was taken. A sample listing is provided in the image below -
Using this data, it would be convenient to understand those objects which are growing in size as the application is used more and more. The way we pin point this objects which are continuously growing in size is by taking multiple heap dump snapshots at contiguous intervals of time and then using MAT to understand their size growth patterns.
One such example is shown in the figure below -
The API library in JDO framework can be used to configure the behavior of JDO during database transactions. Query results Cache:
The query results hold strong references to the retrieved objects. If a query returns too many objects it can lead to OutOfMemory error. Hence, it is advised to use a “Paging” mechanism while retrieving result sets from database.
Paging mechanism of retrieving objects from the database helps in retrieving only a certain number of records from the database instead of retrieving all the records in a single shot. The results retrieved due to this mechanism can be displayed to the end user, providing a link which says, next few records / next page in the application.
When the end user wishes to see more records for the executed query, this link is clicked, which in turn retrieves few more records from the database and displays it in the second page. If all the records wish to be seen, then this process needs to be repeated continuously till all the records are retrieved for the executed query.
The advantage of using this approach is many –
- Retrieve only a fixed number of records regardless of the total number of records being very high. This helps in limiting the application memory consumption because not all the records (hence the objects) are cached in memory.
- Query execution response time is improved significantly because only a few records are retrieved from the database each time.
- User readability improves considerably. It tends to become very difficult if all the records are shown together on the same page of the application. The end user may be searching for only one record amongst all that have appeared, in which case, he has to search through a big list. Paging helps in ease of use of application and readability of results appearing as a consequence of query executions
Ensure best practices are adopted while developing any application. These best practices include both in terms of structuring your code or in terms of ensuring that there are no open resources / handles left unclosed. Developers should ensure that optimized techniques and methods are adopted while building functionality into the application.
Profiling the application periodically at significant stages of application development will help in determining the “code-wellness” and prepare the application for behaving optimally at significant user load / prevent unexpected memory leaks.