Monitoring CICS JVM servers using the IBM Health Center and CICS Explorer
Phil_Wakelin 20000028VV Comment (1) Visits (526)
This article shows how the IBM Health Center can be easily installed into the CICS Explorer and used to create customized perspectives for monitoring CICS JVM servers. Together these two tools provide a powerful platform for profiling CICS Java applications, monitoring memory usage and garbage collection, and performing further diagnostic commands.
To follow the setup in this article you will require the following minimum software requirements:
The configuration in use in this article was as follows:
What is the IBM Health Center?
The IBM Health Center is a diagnostic tool for monitoring the status of a Java Virtual Machine (JVM) and can be used with any IBM JVM including CICS JVM servers. It has a low runtime overhead and provides a highly customizable client as part of the IBM Support Assistant (ISA) workbench. Since the Health Center client is Eclipse based, it can also be installed into other Eclipse environments such as the IBM CICS Explorer.
Installing the Health Center
The first step is to install the Health Center plugin into your CICS Explorer Rich Client Platform (RCP). To do this you will need to add a new software repository to your CICS Explorer, as follows:
Configuring the JVM for monitoring
To enable a JVM for monitoring it is necessary to add a few JVM arguments to enable the Health Center agent and the TCP/IP port it will listen on. When using a CICS JVM server the JVM arguments are set in the JVM profile, which is located in the USS directory specified by the CICS JVMPROFILEDIR SIT parameter.
This will open a new Eclipse Perspective, which is a customizable set of views and includes the CICS Explorer z/OS UNIX Files view for editing zFS files. However, before you can access z/OS you will first need to define an FTP connection to your z/OS system. To do this in the Host Connections view expand z/OS -> z/OS FTP and click Add.
Health centre also uses an additional listening port for its IIOP connection. If you want to control which ports are used ahead of time then you can add the following system property which sets the additional IIOP port used by the Health Center.
Once the JVM server is started the Health Center agent will initialize and immediately start buffering monitoring data ready for collection by the Health Center client. At this stage you can check the Health Center agent is listening on the TCP/IP port using the following MVS TSO command
NETSTAT (PORT 8125
In addition if the JVM server stderr file is viewed, this will show log messages similar to the following indicating the Health Center agent was successfully started and is listening on the configured ports.
Dec 31, 2013 1:16:48 PM com.
Connecting the Health Center client to the JVM
Now that the JVM server is running you can use the Health Center views to monitor the CICS JVM server. The Health Center Environment perspective provides a convenient way to access all the Health Center function. It can be simply accessed using Windows -> Open Perspective -> Other -> Health Center Environment
The Status view in this perspective can now be used to access the different functions of the Health Center, such as CPU, Environment, Garbage Collection, Locking, Native Memory, Profiling and Threads. The next section will discuss how some of these can be used to analyze the CICS JVM server environment.
The Garbage Collection perspective provides a set of views to assist in analyzing the garbage collection (GC) process used by the JVM to manage memory in the JVM heap. Using the default gencon GC policy splits the Java heap into two areas, the new or nursery area, and the old or tenured area. CICS JVM server statistics call new or nursery activity minor GC and call old or tenured activity major GC. The Summary view provides detailed information on the GC process, however, much of this information is also displayed in the CICS Explorer JVM Servers view, which can be added to the Garbage Collection perspective using Window -> Show view -> Other –> JVM Servers. In addition individual Health Center graphs for Used heap and Pause time can also be added using the same Show view menu, and a customized perspective created with a combination of both Health Center and CICS views as shown below.
Custom perspectives such as these can be saved for future use by right-clicking on the current perspective and selecting Save As, which will add the perspective to the list of available perspectives on the top row.
In our example above, you can see that the initial heap of 30MB provides adequate space for the current memory usage, and the GC process is able to minimize the heap occupancy to approximately 6.8MB each time it is invoked. Out of the 949 garbage collections events that have run, all but 5 were in the nursery area (minor), which indicates most garbage is short lived objects from the CICS transaction invocations.
One of the most useful GC views provided by the Health Center is the Analysis and Recommendations view. This provides analysis for each perspective, and is a simple way of analyzing heap usage. In the Garbage Collection perspective above, the following analysis was provided, which relates accurately to the increase in workload used to generate this test.
The Profiling perspective allows method level profiling of the applications running within the JVM. The methods can be filtered by class or package name. In the figure below we can see that during 23.1% of the samples taken, the method that was running was called from the main method of the JavaLinker class which is defined as the entry point for our CICS program. In addition the formatTime() method was active in 3 samples out of 44, and so could be considered as a candidate for optimization.
Additionally the Invocation and Called method views allows you to analyze the call path of each profiled method to ascertain how it was invoked, and what further methods it calls.
The Threads perspective provides detailed analysis of all the threads running within the JVM. The Current threads view supplies a useful Thread name filter box, which allows the display to be filtered. Using a filter of *.TASK.* will locate the CICS task threads which are dubbed with the pattern prog
Customizing data collection
Instead of requiring a socket connection to a monitored JVM, Health Center can save the data that it is analyzing to a number of .hcd files on the local machine. These files can then be opened in the Health Center Eclipse client at a later date, without the need for a live connection. This is termed headless mode, and is useful for systems where you cannot connect a client to the agent. For example, where firewall restrictions prevent connection.
To enable headless mode set the following property in the CICS JVM profile.
To control the directory location of the temporary files used to buffer data for headless collection you can set the following system property. In the example below this is set to a a directory unique to the specific CICS JVM server using the &APPLID and &JMVSERVER substitution symbols which will be substituted at runtime by the relevant CICS values.
Additionally you should also set the run duration (in minutes) and a number of runs, this will enable you to collect output files after a set period of time.
In this directory you will see a set of files created as shown below, and then after the duration has expired a .hcd suffixed file will be created.
$> ls /cic
To analyze the run transfer the .hcd suffixed file to your workstation in binary mode and open using the File -> Load Data menu in the Health Center client, which will then enable analysis of the run using any of the facilities described.
If the data that Health Center collects exceeds the maximum file size of 2GB the agent automatically creates a second file, and so on. If you are gathering lots of data, a long headless run will result in a collection of files. By default, Health Center keeps only the last 10 files. You can override this value by using the following system property.
Alternatively to minimize data collection in a JVM that is not being monitored you can start the agent but delay collection until a client connects. This means there is no impact on the application that is running, until the client actually connects to the agent. To enable this mode set the following JVM property.
This article has shown how the functionality of CICS Explorer for monitoring Java workloads in CICS TS, can be easily extended using the IBM Health Center plugin. Let us know how you get on and what further information you are interested in.
If you wish to explore this area further the following articles and papers may be useful references: