Monitor a Java application with the Health Center API, Part 2

Add a method-profiling view to the deadlock detection application

The 2.1 release of Health Center contains a powerful API. The API allows Java developers to embed Health Center in their applications and harness its monitoring power to troubleshoot problems. Part 2 of this series takes the deadlock detection application from Part 1 and adds a method-profiling view to show where the application spends most of its CPU cycles.

Share:

Toby Corbin, Software Engineer, IBM  

Toby CorbinToby Corbin is lead developer of the Health Center monitoring tool; he also develops other consumability tooling within the IBM Java Technology Centre. He joined IBM in 2001 and, for four years, developed national language support and globalization of the Java Runtime Environment. Toby developed the Swing and AWT libraries for two years.



26 February 2013

Also available in Chinese Russian Japanese Vietnamese Spanish

The IBM® Monitoring and Diagnostics tools for Java - Health Center (Health Center) is a free, low-overhead diagnostic tool and API for monitoring an application that runs on an IBM Java virtual machine (JVM). For specifics on what the API can do, see Part 1. In this article, take the deadlock detection application that you developed in Part 1 and add a method-profiling view to show where your application spends most of its CPU cycles. (See Download to get the full source code for the examples.)

System requirements

The Health Center API bundle requires a minimum of Eclipse 3.4 or Eclipse 4.x.

Testing against the application

In Part 1, you tested against an application, which generated a deadlock condition. For this article, use a modified version of that application, which contains some functions that stresses the CPU. See Download for the source code. Listing 1 shows an example fragment:

Listing 1. Fragment of new source code for GenerateDeadLock
private class runSlowMethods extends Thread {

   public void run() {
      SlowClassAndMethod1 sCAM1 = new SlowClassAndMethod1();
      SlowClassAndMethod2 sCAM2 = new SlowClassAndMethod2();

      sCAM1.start();
      sCAM2.start();
   }

   private class SlowClassAndMethod1 extends Thread {

      public void run() {
         while (true) {
            slowMethod1();
            try {
               Thread.sleep(2000);
            } catch (InterruptedException e) {
               e.printStackTrace();
            }
         }
      }

      private void slowMethod1() {
         String largeString = new String("a string to add");
         for (int i = 0; i < 1000000; i++) {
            largeString.concat(largeString);
         }

      }
   }

   private class SlowClassAndMethod2 extends Thread {

      public void run() {
         while (true) {
            slowMethod2();
            try {
               Thread.sleep(2000);
            } catch (InterruptedException e) {
               e.printStackTrace();
            }
         }
      }

      private void slowMethod2() {
         HashMap map = new HashMap();
         String largeString = new String("another string to add");
         for (int i = 0; i < 1000000; i++) {
            map.put(i, largeString);
            largeString.concat(largeString);
         }

      }

Start this program with the Health Center agent attached to it. To start the application with the agent using Java 5 SR10 and later, Java 6 SR5 and later, and Java 7, use this command (see Figure 1):

java -Xhealthcenter GenerateDeadlock
Figure 1. Starting the application
Screen capture of the commandline software to install

To start the application with the agent using Java 5 SR9 and earlier, or Java 6 SR4 and earlier, use:

java -agentlib:healthcenter -Xtrace:output=healthcenter.out GenerateDeadlock

You can download the IBM developer kits.

Coding the enhanced monitoring application

To modify the DeadlockDemo application you created in Part 1, open the Application.java file (see Figure 2):

Figure 2. Open Application.java
Screen capture of plug-in application.java

Replace all the code in Application.java with the source code (see Download). Listing 2 shows an example fragment:

Listing 2. Fragment of replacement code for Application.java
private class ProfileApplication extends Thread {

   String methodTree = null;

   public void run() {

      try {
         Thread.sleep(10000);
      } catch (InterruptedException e) {
         e.printStackTrace();
      }

      while (keepRunning) {
         analyzeMethods();
         try {
            Thread.sleep(2000);
         } catch (InterruptedException e) {
            e.printStackTrace();
         }
      }
   }

   private void analyzeMethods() {
      ProfilingData profilingData;
      profilingData = hcMon.getProfilingData();

      if (profilingData == null) {
         return;
      }

      MethodProfileData[] mPD = profilingData.getProfilingEvents();
      if (mPD != null) {

         final SimpleProfileData[] sPD = new SimpleProfileData[mPD.length];
         int index = 0;
         for (MethodProfileData mP : mPD) {
            methodTree = new String();
            sPD[index] = new SimpleProfileData();
            sPD[index].setCount(mP.getMethodSampleCount());
            sPD[index].setMethodName(mP.getMethodName());
            MethodProfilingNode[] mPN = mP.getCallingMethods();
            for (MethodProfilingNode node : mPN) {
               if (node instanceof MethodProfilingNode) {
                  walkProfileTree(node);
               }
            }
            sPD[index].setMethodTree(methodTree);
            index++;
         }

         Arrays.sort(sPD);

         display.asyncExec(new Runnable() {
            public void run() {

               profileText.setText("method: "
                     + sPD[0].getMethodName()
                     + "\n		sample count:  "
                     + sPD[0].getCount() + "\n"
                     + sPD[0].getMethodName()
                     + sPD[0].getMethodTree());
               shell.redraw();
            }
         });

      }
   }

This code is different from the original, but most of the changes are to set up displays and apply the correct behavior for the application. The actual Health Center API code just has more calls for the Method profile data, which you'll see next.


Profiling the data

Health Center uses a sample-based profiler, which gives you a sampled view of all the methods that your application is using. This profiler is useful for identifying performance bottlenecks because your application might spend too much time in a single method, causing the performance problem.

To access the profiling data, use a similar call similar to access the threads data:

ProfilingData profilingData = HealthCenter.getProfilingData();

The ProfilingData class has methods that give access to all the methods sampled, their sample count, and the call hierarchy.

The method getProfilingEvents() returns an array of MethodProfileData, which has an entry for every method sampled. For each sampled method, you can get the sample count with getMethodSampleCount(), the method name with getMethodName(), and the methods that called it with getCallingMethods().

Use those methods to generate the call hierarchy of the most sampled method. Then, use that information to fine-tune the performance of the monitored application.


Monitoring for deadlocks and method profiling

Now you have the new deadlock and profiling monitoring application with the revised deadlock infected program is running in the background. Run the deadlock and profiling monitoring application to see "Deadlock detected" messages on the Application monitoring window (see Figure 3):

Figure 3. Monitoring for deadlocks
Screen capture of deadlocks in application monitoring

Clearly, you have data. Now it's time to figure out what it means.


What the profiling data shows

In the top panel, the sample count increases as the application runs, showing that this method consumes the most CPU cycles. Listing 3 shows the output of only the top method:

Listing 3. Outputting the top method
profileText.setText("method: "
   + sPD[0].getMethodName()
   + "\n		sample count:  "
   + sPD[0].getCount() + "\n"
   + sPD[0].getMethodName()
   + sPD[0].getMethodTree());

But you can change it to output more methods by looping through all the elements in the sPD array.

When the profiler runs in Health Center, a method gets a count if it actively runs Java code on the stack. The more samples a method has, the more work it does. Sample count might not reflect how often a method is called. It is an indication of how much work that method does in relation to the other methods in the application.

In this example, you can see that the method java.util.HashMap.rehash(int) is the most sampled. Its call stack (where it is called from) shows that it originates from a call in GenerateDeadlock$runSlowMethods$SlowClassAndMethod2.run().

Looking at the next call in the hierarchy, the run() method calls GenerateDeadlock$runSlowMethods$SlowClassAndMethod2.slowMethod2(), which is responsible for the call to java.util.HashMap.put. That, in turn, ultimately results in the call to rehash().

If you look at the slowMethod2() method in the deadlock application, you can see a loop that puts entries into a HashMap (see Listing 4):

Listing 4. Loop that puts entries into a HashMap
private void slowMethod2() {
   HashMap map = new HashMap();
   String largeString = new String("another string to add");
   for (int i = 0; i < 1000000; i++) {
      map.put(i, largeString);
      largeString.concat(largeString);
   }
}

You can comment out that loop in the method, as in Listing 5:

Listing 5. Commenting out the loop
   private void slowMethod2() {
      HashMap map = new HashMap();
      String largeString = new String("another string to add");
//      for (int i = 0; i < 1000000; i++) {
//         map.put(i, largeString);
//         largeString.concat(largeString);
//      }
   }

If you rerun the test application and then the new deadlock and profiling tool, notice that the problem was fixed. A new method appears in the profiling panel (see Figure 4):

Figure 4. Problem fixed—new method appears
Screen capture of a new method in profiling panel

This process—that identifies hot methods and helps to resolve performance bottlenecks—can now start again.


Conclusion

These articles show the basics of how to get started with the Health Center API. You can apply these techniques to extract and use any data inside Health Center. For example, you might use the API to get details on WebSphere® methods only when they start to go hot, or to trigger a system dump when the application looks likes it is running out of memory.


Download

DescriptionNameSize
Source code for articlej-healthcareapi2-source.zip3KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Java technology on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java technology
ArticleID=859515
ArticleTitle=Monitor a Java application with the Health Center API, Part 2
publish-date=02262013