Troubleshooting
Problem
If your Java application experiences high processor usage on an AIX® operating system, there are a number of diagnostic data files that are useful for diagnosing the problem. This mustgather document describes about the diagnostic data files to be collected and the procedure to collect the same..
Resolving The Problem
The following set of diagnostic data files need to be collected manually during the time of high processor usage by your java application:
1. Javacores
2. Core dumps
3. Verbose GC logs(native_stderr.log)
4. Process logs (ps)
5. Paging usage logs (vmstat)
6. CPU Profiling logs (Tprof)
7. Netstat logs
For these diagnostic data files to be created, check that the system is configured correctly as per setup document.
Please collect the diagnostic data files manually during the time of high processor usage as per the below steps:
1. Generate at least three javacores and core dumps using the following command:
kill -3 [PID_of_problem_JVM]
There needs to be 2 or 3 minutes time interval in between each set of dumps(javacore and core dump) generated. Otherwise spread the time interval across the issue time and generate the above set of diagnostic data files at each of the points below:
a) As soon as possible after the application starts.
b) When you notice a definite increase in processor usage, based on observation or monitoring tools.
c) When processor usage peaks and remains consistently high, based on observation or monitoring tools.
Please note that all the javacores and core dumps generated can be found in the current path and the names of them will follow the below format:
Javacore : javacore.<time stamp>.<id>.txt
Core dump : core.<time stamp>.<id>.dmp
Run the JExtract tool against the generated core dump file using the following command:
<installpath>/jre/bin/jextract [coredump]
For 64bit JVM enabled with -Xcompressedrefs option, please use
<installpath>/jre/bin/jextract -J-Xcompressedrefs [coredump]
Save the resulting archive file, which has a name in the format of
core.<time stamp>.<id>.dmp.zip
2. In case if the core dump files are not generated as per above step, another method for generating the core dump file is by using AIX® gencore utility.
In a separate terminal, run the following command:
gencore <process-id>
This command creates a dump file named:
core.<time stamp>.<id>.dmp
Run the JExtract tool against the generated core dump file using the following command:
<installpath>/jre/bin/jextract [coredump]
For 64bit JVM enabled with -Xcompressedrefs option, please use
<installpath>/jre/bin/jextract -J-Xcompressedrefs [coredump]
Save the resulting archive file, which has a name in the format of
core.<time stamp>.<id>.dmp.zip
Rename the core dump file generated to something unique before generating the next dump file, otherwise you might lose useful information from the dump file.
3. Collect Verbose Garbage Collector data. This data is in the location specified during setup. Alternatively, the data is sent to the stderr output.
4. Obtain a current listing of the processes running on the system, by using the following command:
ps -efH > ps_efh.txt
5. Collect the process size monitoring data. The data is in the location specified during setup.
6. Collect the paging size usage data. The data is in the location specified during setup
7. Run the tprof tool and profile the java application during the time of high processor usage.The tprof profile data is written to the location specified during setup, and is stored in below path and file names.
./tprof_ps/ps-w-tprof.log
./tprof_ps/sleep.prof
8. Collect the netstat log containing the snapshot of network activities, by using the following command:
netstat -an > netstat1.out
Please collect multiple netstat logs during the time of high processor usage.
After collecting all the above diagnostic data files, you can submit them for help with diagnosing the problem.
Related Information
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21222484