Determining the CPU usage of applications running in WebSphere Application Server on z/OS

When many applications are deployed and running concurrently in an IBM® WebSphere® Application Server environment, it is often desirable to determine which of these applications are consuming CPU resources. This is especially true if it is suspected that one or more applications are engaged in more CPU usage than is expected or sustainable. This article shows how to utilize z/OS® facilities to identify the CPU resource consumption of individual WebSphere Application Server applications. This content is part of the IBM WebSphere Developer Technical Journal.

Karl Gaffney, Software Consultant, IBM

author photoKarl Gaffney is a Software Consultant in the IBM WebSphere Accelerated Value Pro-gram (AVP) Europe, and is based at the IBM Hursley Lab in the U.K. He holds a PhD in Artificial Intelligence andl has extensive experience as a software test engineer testing a wide range of functionality of WebSphere Application Server across a variety of plat-forms including z/OS.



Daniel McGinnes (mcginnes@uk.ibm.com), Performance Specialist, IBM

Photo of Daniel McGinnesDaniel McGinnes a Performance Specialist at the IBM Hursley Lab in the U.K. He works on measuring and tuning performance of WebSphere Operational Decision Management. He has previously worked on the performance of other WebSphere portfolio products, including WebSphere Business Events, WebSphere Application Server and WebSphere Enterprise Service Bus. He has also worked as an IT specialist providing consulting to IBM customers across EMEA and USA.



05 December 2012

Also available in Chinese

Introduction

It is common for IBM WebSphere Application Server environments to host multiple enterprise applications of varying types that are underpinned by different Java™ EE technologies. These applications typically run concurrently and each one consumes system resources, such as CPU cycles and memory, relative to the amount of work they are doing. It is often the case during both performance testing and production usage that it would be useful to view the resources consumed on a per application basis, rather than just for the application server as a whole.

There are many tools available to monitor performance statistics within WebSphere Application Server. A primary example is the WebSphere Application Server Performance Monitoring Infrastructure (PMI). This enables you to monitor such key areas as thread pools and connection pools, which can give an indication as to which applications are consuming CPU resources, especially if there are very few applications running. However, if there are many applications deployed and running in a server it can be very challenging to ascertain exactly which ones are consuming the system resources.

On z/OS there are system tools available via the Service Management Facility (SMF) and Resource Management Facility (RMF) that can help the WebSphere Application Server system administrator obtain information on the amount of CPU usage occurring for each individual application running within the server. Any "problem applications" discovered can then be investigated individually to establish the underlying reasons for the performance issues.

This article is aimed at z/OS administrators and explains how to ascertain CPU usage for individual applications running in WebSphere Application Server V6.x to WebSphere Application Server V8.5 on z/OS by utilizing z/OS classification rules, SMF, and RMF reports.


Example environment

The applications described here are hypothetical and referred to only for illustrative purposes; code is not provided. In a real-world environment the relevant business applications running in the environment will yield the performance data for subsequent analysis, and the steps described in this article should be applied for those applications.

For the purpose of this article, assume four applications are deployed into WebSphere Application Server on z/OS that will be subjected to varying concurrent workloads:

  • Two are simple web applications running in the web container.
  • One is an Enterprise JavaBean (EJB) application.
  • One is a message-driven bean (MDB) based application using the WebSphere Application Server Service Integration Bus (SIB).

This setup could easily be scaled up to include however many applications the you wish. The applications have very limited function in this environment except to consume CPU cycles via mathematical operations when they are invoked, and are chosen to cover three main Java EE technology areas.


Classification rules XML

In order to separate the different applications into different report classes that will be shown in an RMF report, the first step is to create a classification rules XML document. For this example, an XML document called zwlm.xml (this document can be given any name) is created to define classification rules for each application and to link these classification rules to transaction classes that are subsequently created in the z/OS workload manager (WLM). This is shown in Listing 1.

Listing 1. Classification rules XML document
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Classification SYSTEM "Classification.dtd" >
<Classification schema_version="1.0">
  <!--
       HTTP Classification Rules
  -->
  <InboundClassification type="http"
                         schema_version="1.0"
                         default_transaction_class="TC_1">
    <http_classification_info transaction_class="TC_2"
                              uri="/MigrateHelloWorldWAR/*"
                              description = "HW1" />
    <http_classification_info transaction_class="TC_3"
                              uri="/MigrateHelloWorldWAR2/*"
                              description="HW2" />
  </InboundClassification>

  <!--
        IIOP Classification Rules
  -->
  <InboundClassification type="iiop" 
                         schema_version="1.0"
                         default_transaction_class="TC_4">
    <iiop_classification_info transaction_class="TC_5"
                              application_name="TestEJBEAR"
                              module_name="TestEJB.jar"
                              description="Sample EJB Classification" />
  </InboundClassification>

  <!-- 
     MDB Classification Rules
  -->
  <SibClassification type="jmsra" 
                     schema_version="1.0"
                     default_transaction_class="TC_8">
     <sib_classification_info transaction_class="TC_9"
            bus="BUS1"
            destination="ASSURE1" description="SIB MDB" />
   </SibClassification>

</Classification>

As you can see, this document contains three sections, one for each classification rule type: http for web applications, iiop for EJB applications, and jmsra for SIB MDB based applications. The default_transaction_class fields are assigned values that will be used as defaults for that type of work. For more specific HTTP classification rules, the uri field holds the context root of the deployed WAR. For example, the value:

uri="/MigrateHelloWorldWAR/*"

ensures that the transaction class TC_2 will be invoked whenever that context root is used, such as:

http://myserver.hursley.ibm.com:9080/MigrateHelloWorldWAR/MyServlet?TIME=10000

With IIOP (EJB) classification rules, the application (EAR) name and module name (JAR) must be provided.

With SIB MDB applications, the SIB name and the destination that the MDB is “listening” to is provided. Typically, an individual MDB will listen to its own destination. This means that an individual application's CPU usage can be readily determined.

The full Classification.dtd associated with the XML is shown in Listing 2.

Listing 2. Classification rules DTD document
<?xml version='1.0' encoding="UTF-8"?>
<!ELEMENT Classification (InboundClassification|SibClassification)+>
<!ATTLIST Classification schema_version CDATA #REQUIRED>
<!ELEMENT InboundClassification ((iiop_classification_info*
	|http_classification_info*|endpoint*))>
<!ATTLIST InboundClassification type (iiop|mdb|http) #REQUIRED>
<!ATTLIST InboundClassification default_transaction_class CDATA #REQUIRED>
<!ATTLIST InboundClassification schema_version CDATA #REQUIRED>
<!ELEMENT iiop_classification_info (iiop_classification_info*)>
<!ATTLIST iiop_classification_info activity_workload_classification CDATA #IMPLIED>
<!ATTLIST iiop_classification_info application_name CDATA #IMPLIED>
<!ATTLIST iiop_classification_info component_name CDATA #IMPLIED>
<!ATTLIST iiop_classification_info description CDATA #IMPLIED>
<!ATTLIST iiop_classification_info method_name CDATA #IMPLIED>
<!ATTLIST iiop_classification_info module_name CDATA #IMPLIED>
<!ATTLIST iiop_classification_info transaction_class CDATA #REQUIRED>
<!ELEMENT endpoint (classificationentry*)>
<!ATTLIST endpoint defaultclassification CDATA #REQUIRED>
<!ATTLIST endpoint name CDATA #REQUIRED>
<!ATTLIST endpoint type (messagelistenerport) #REQUIRED>
<!ATTLIST endpoint description CDATA #IMPLIED>
<!ELEMENT classificationentry EMPTY>
<!ATTLIST classificationentry classification CDATA #REQUIRED>
<!ATTLIST classificationentry selector CDATA #REQUIRED>
<!ATTLIST classificationentry description CDATA #IMPLIED>
<!ELEMENT http_classification_info (http_classification_info*)>
<!ATTLIST http_classification_info host CDATA #IMPLIED>
<!ATTLIST http_classification_info port CDATA #IMPLIED>
<!ATTLIST http_classification_info uri CDATA #IMPLIED>
<!ATTLIST http_classification_info description CDATA #IMPLIED>
<!ATTLIST http_classification_info transaction_class CDATA #REQUIRED>
<!ELEMENT SibClassification (sib_classification_info+)>
<!ATTLIST SibClassification type (jmsra|destinationmediation) #REQUIRED>
<!ATTLIST SibClassification default_transaction_class CDATA #REQUIRED>
<!ATTLIST SibClassification schema_version CDATA #REQUIRED>
<!ELEMENT sib_classification_info EMPTY>
<!ATTLIST sib_classification_info transaction_class CDATA #REQUIRED>
<!ATTLIST sib_classification_info selector CDATA #IMPLIED>
<!ATTLIST sib_classification_info bus CDATA #IMPLIED>
<!ATTLIST sib_classification_info destination CDATA #IMPLIED>
<!ATTLIST sib_classification_info discriminator CDATA #IMPLIED>
<!ATTLIST sib_classification_info description CDATA #IMPLIED>

It is convenient to create a directory called /classification in the relevant application server profile on the HFS file system. Both the DTD and the XML are uploaded (for example, via ftp in binary) to this location. (See Resources for more information.)

A WebSphere Application Server environment variable called wlm_classification_file is defined to reference the classification rules XML document. This variable can be set at the cell, node, or server level. If the cell or node level is specified, the information contained in it must be accessible and applicable to all servers that inherit the specification from that cell or node.

In the administrative console navigate to Environment > WebSphere variables > New. The Name of the new variable is defined as wlm_classification_file and the Value similar to (for example) /WebSphere/Base/AppServer/profiles/default/classification/zwlm.xml (Figure 1).

Figure 1. Setting the wlm_classification_file environment variable
Figure 1. Setting the wlm_classification_file environment variable

Creating WLM classification rules

From the XML definition (Listing 1), you could see that seven transaction classes are specified, arbitrarily named: TC_1, TC_2, TC_3, TC_4, TC_5, TC_8, and TC_9. Seven WLM classification rules need to be created to utilize these classes. To do this:

  1. Start the z/OS WLM application according to the the installation instructions (typically the iwmarin0 command is executed).
  2. On the Choose Service Definition screen, select the Extract definition from WLM couple data set (Figure 2).
    Figure 2. Extract service definition for defining classification rules
    Figure 2. Extract service definition for defining classification rules
  3. Select the Classification Rules option (Figure 3).
    Figure 3. Classification rules are then selected using option 6
    Figure 3. Classification rules are then selected using option 6
    If there already exists a CB type service class for WebSphere Application Server classification, as shown in Figure 4, then this can simply be modified by selecting option 3 against it.
    Figure 4. Select option 3 to modify the existing WebSphere classification service class
    Figure 4. Select option 3 to modify the existing WebSphere classification service class
    If a type CB classification rules service class for WebSphere Application Server does not exist, then it will be necessary to create one using option 1. Some examples of creating these classes for WebSphere Application Server are available in Defining Component Broker (CB) Classification Rules.
  4. When this WebSphere Application Server classification service class has been selected, classification rules are inserted. In Figures 5 and 5a, option I, Insert rule, is selected repeatedly to enter each classification rule. Each transaction class specified above in Listing 1 should have its own classification rule. The transaction rule name corresponds exactly to the Qualifier name of each rule. Each rule should have its own report class with a name, while arbitrary, should have some affinity with the transaction classes in the rules.

    For example, the report class and transaction class could share the same numerical suffix. In this test, the report classes are defined as KGREP1, KGREP2, and so on. The Type of the rule is TC (Transaction Class), and the Service Class field is left blank to be inherited from the (in this case) CBCLASS.

    Figure 5. Defining classification rules for each transaction class
    Figure 5. Defining classification rules for each transaction class
    Figure 5a. Defining classification rules for each transaction class, continued
    Figure 5. Defining classification rules for each transaction class
  5. Notice that TC_6, TC_7 and TC_10 in Figures 5 and 6 are not actually used as part of this test. WCL* is a pre-existing rule that is also not used during this test. When ENTER is pressed for the above screen, the user is prompted to create the report classes specified. This is accepted with another ENTER as shown in Figures 6 and 6a below.
    Figure 6. Creating the report classes
    Figure 6. Creating the report classes
    Figure 6a. Creating the report classes, continued
    Figure 6. Creating the report classes
  6. If ENTER is pressed, followed by the function key F3 (to exit), confirmation is given that the type CB subsystem has been successfully modified (Figure 7).
    Figure 7. Successfully exiting the creation of the classification rules
    Figure 7. Successfully exiting the creation of the classification rules
  7. It is now necessary to install and activate the WLM definition that has been altered. Press the F3 function key to exit the Subsystem Type Selection List, and the console returns to the WLM definition menu.
  8. Place the cursor over the Utilities menu bar, and press ENTER (Figure 8).
  9. Select option 1, Install definition and press ENTER.
    Figure 8. Install the changes made to the configuration
    Figure 8. Install the changes made to the configuration
    The message Service definition was installed. (IWMAM038) should now display.
  10. The service policy now needs to be activated so that it will be applied. To achieve this, place the cursor over the Utilities menu and press ENTER. this time, however, select option 3, Activate service policy. Select the policy (Figure 9) and press ENTER.
    Figure 9. Activating the service policy
    Figure 9. Activating the service policy
    Figure 9 shows there is only one service policy in this example environment and it is selected for activation. The message Service policy NORMAL was activated. (IWMAM060) should now display.

Creating the post-processing JCL

In order to post-process the SMF data collected during execution, some JCL is required to perform two functions:

  1. Dump the SMF records. These will contain the RMF report data (including CPU usage) that is being collected about the WebSphere Application Server applications.
  2. Generate the RMF report based on this data dump.

The RMF report can then be collected and analyzed for WebSphere Application Server application CPU usage.

The JCL used to obtain the RMF reports is shown in Listing 3 .

Listing 3. JCL to obtain RMF reports
//RMFRPT  JOB MSGCLASS=H,MSGLEVEL=(1,1),REGION=0M
/*JOBPARM SYSAFF=MV60
//SMFDUMP  EXEC PGM=IFASMFDP
//DUMPINA  DD   DSN=SYS1.MV60.MANC,DISP=SHR,AMP=('BUFSP=65536')
//DUMPINB  DD   DSN=SYS1.MV60.MAND,DISP=SHR,AMP=('BUFSP=65536')
//DUMPOUT  DD   DSN=&&SMF89,DISP=(NEW,PASS),SPACE=(CYL,(10,10))
//SYSPRINT DD   SYSOUT=*
//SYSIN  DD *
  INDD(DUMPINA,OPTIONS(DUMP))
  INDD(DUMPINB,OPTIONS(DUMP))
  OUTDD(DUMPOUT,TYPE(30,70,71,72,73,74,75,76,77,78,79))
  START(1600)
  END(1800)
/*
//RMFDTL EXEC PGM=ERBRMFPP,REGION=0M
//SYSPRINT DD SYSOUT=*
//MFPINPUT DD DSN=*.SMFDUMP.DUMPOUT,DISP=(SHR,PASS)
//SYSIN    DD *
  REPORTS(CPU,HTTP,IOQ,OMVS,PAGESP,PAGING,VSTOR(D))
  REPORTS(DEVICE(DASD,COMM),XCF,CHANNEL,HFS)
  REPORTS(PAGING)
  REPORTS(PAGESP)
  REPORTS(CACHE(SUBSYS))
  SYSRPTS(WLMGL(SCLASS,SCPER,RCLASS,RCPER,POLICY))
  SUMMARY(INT,TOT)
/*
//

Notice the period during which these reports will be dumped and generated is defined in the START and END parameters (between 16:00 and 18:00 hours in this sample). The report Interval is 30 minutes by default, so reports are generated for 30 minute periods. This parameter can be changed to modify the default report interval by using the INTVAL parameter in the SMFPRM member of the SYS1.PARMLIB dataset. (See the Information Center for more information.)


Obtaining the performance statistics

When the WebSphere Application Server server is restarted after the XML is uploaded there should be evidence of this file being successfully recognized by WebSphere Application Server in the z/OS job logs, specifically in the control region logs. You should see a message similar to this:

The /WebSphere/V80S01S1/AppServer/classification/zwlm.xml workload classification file was loaded at 2 ...

If there is a problem with the XML (for example, if it is structured incorrectly or contains typographical errors) then a message similar to this will be seen in the logs, indicating that the XML file should be analyzed:

Problems parsing WLM classification XML file ...

To obtain the performance statistics:

  1. In the dataset member RMFRPT, the START and END variables are changed to encompass the period required. In this example, START and END were changed to 1000 and 1300 respectively. No submission of the JCL or a WebSphere Application Server restart is required for this.
  2. To obtain some statistical information for this test, the WebSphere Application Server applications were subjected to varying workloads between 11:10 and 11:25.
  3. At 11:35 the JCL was submitted as shown in Figure 10. Because reports are generated (by default) for half hour periods from hh:00 to hh:30 and hh:30 to hh:00, submitting after 11:30 ensures that all relevant statistics are captured from 11:00 to 11:30.
Figure 10. Submitting the JCL
Figure 9. Activating the service policy

As a result of this, the report job RMFRPT is SDSF can be seen on the PRINT queue by issuing the ST command.

Figure 11. RMF report on the z/OS print queue
Figure 11. RMF report on the z/OS print queue

Expanding RMFRPT, as shown in Figure 11, displays all the subcomponents of this job, shown in Figure 12.

Figure 12. Subcomponents of the RMFRPT job on the print queue
Figure 12. Subcomponents of the RMFRPT job on the print queue

PPXSRPTS contains the workload activity report that is needed and can be downloaded for viewing in the usual manner. The other subcomponents contain other RMF reports that may also be of interest, but are outside the scope of this article.


Viewing the statistics

PPXSRPTS is downloaded into a file called REPORT.txt (Figure 13). This document contains workload activity for half-hourly segments for the period defined in your JCL (10:00 onwards in this example). The relevant section here is from 11:00 to 11:30 because that is when workload occurred with the WebSphere Application Server applications. This workload was heavily biased towards the web applications and these were required to do far more mathematical calculations (as a simple mechanism to consume CPU cycles) than the MDB and EJB based ones.

Figure 13. Application workload activity (REPORT.txt)
Figure 13. Application workload activity (REPORT.txt)

From Listing 1 and Figure 5 it is known that KGREP2 and KGREP3 are the report classes associated with the two web applications; KGREP5 is associated with the EJB based application and KGREP9 with the MDB based application.

The report shows a range of data that can be used to understand the performance characteristics of each application. Some of the most useful are:

  • ENDED: Number of transactions that completed within the period.
  • ENDED/S: Average transaction rate (ended per second) for this period.
  • ACTUAL: Average elapsed time for each transaction.
  • EXECUTION: Average elapsed time spent executing the transaction.
  • QUEUED: Average elapsed time spent on WLM queue waiting to be executed.
  • SERVICE TIME – CPU: Time in seconds consumed on all processors.
  • SERVICE TIME – AAP: zAAP service time in seconds.
  • SERVICE TIME – IIP: zIIP service time in seconds.
  • APPL% - CP: % of CPU time used by transactions running on general purpose processors.
  • APPL% - AAPCP: Percentage of CPU time used by zAAP eligible transactions running on general purpose processors (standard CPs). This is a subset of APPL% CP.
  • APPL% - IIPCP: Percentage of CPU time used by zIIP eligible transactions running on general purpose processors. This is a subset of APPL% CP.
  • APPL% - AAP: Percentage of CPU time used by transactions executed on zAAPs in the service or report class period.
  • APPL% - IIP: Percentage of CPU time used by transactions executed on zIIPs in the service or report class period.

See the Information Center for information on other fields in the report.

Using the SERVICE TIME – CPU and APPL% - CP fields, the reports in Figure 13 show the larger amounts of CPU consumed by the web applications (KGREP2 and KGREP3). In a real world scenario where general CPU usage was deemed to be excessive, this technique could pinpoint those applications as a potential cause.

This information can now be used to enable application developers to concentrate efforts on optimizing the applications that are consuming the most resources. Profiling tools such as the IBM Monitoring and Diagnostic Tools for Java - Health Center are available for analyzing problem applications in more detail; further information on the use of these tools is outside the scope of this article.


Conclusion

The CPU usage of individual applications running on IBM WebSphere Application Server for z/OS that utilize different Java EE technologies can be ascertained by using WLM classification rules with a classification rules XML document to define which applications are associated with the transaction classes. With these deployed, Service Management Facility (SMF) data can be collected, enabling the generation of Resource Management Facility (RMF) reports for view the CPU data. This mechanism, as described in this article, can be valuable in determining why z/OS resources are being saturated when a large number of applications co-exist in WebSphere Application Server on a sysplex.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=848506
ArticleTitle=Determining the CPU usage of applications running in WebSphere Application Server on z/OS
publish-date=12052012