It is common for IBM WebSphere Application Server environments to host multiple enterprise applications of varying types that are underpinned by different Java™ EE technologies. These applications typically run concurrently and each one consumes system resources, such as CPU cycles and memory, relative to the amount of work they are doing. It is often the case during both performance testing and production usage that it would be useful to view the resources consumed on a per application basis, rather than just for the application server as a whole.
There are many tools available to monitor performance statistics within WebSphere Application Server. A primary example is the WebSphere Application Server Performance Monitoring Infrastructure (PMI). This enables you to monitor such key areas as thread pools and connection pools, which can give an indication as to which applications are consuming CPU resources, especially if there are very few applications running. However, if there are many applications deployed and running in a server it can be very challenging to ascertain exactly which ones are consuming the system resources.
On z/OS there are system tools available via the Service Management Facility (SMF) and Resource Management Facility (RMF) that can help the WebSphere Application Server system administrator obtain information on the amount of CPU usage occurring for each individual application running within the server. Any "problem applications" discovered can then be investigated individually to establish the underlying reasons for the performance issues.
This article is aimed at z/OS administrators and explains how to ascertain CPU usage for individual applications running in WebSphere Application Server V6.x to WebSphere Application Server V8.5 on z/OS by utilizing z/OS classification rules, SMF, and RMF reports.
For the purpose of this article, assume four applications are deployed into WebSphere Application Server on z/OS that will be subjected to varying concurrent workloads:
- Two are simple web applications running in the web container.
- One is an Enterprise JavaBean (EJB) application.
- One is a message-driven bean (MDB) based application using the WebSphere Application Server Service Integration Bus (SIB).
This setup could easily be scaled up to include however many applications the you wish. The applications have very limited function in this environment except to consume CPU cycles via mathematical operations when they are invoked, and are chosen to cover three main Java EE technology areas.
Classification rules XML
In order to separate the different applications into different report classes that will be shown in an RMF report, the first step is to create a classification rules XML document. For this example, an XML document called zwlm.xml (this document can be given any name) is created to define classification rules for each application and to link these classification rules to transaction classes that are subsequently created in the z/OS workload manager (WLM). This is shown in Listing 1.
Listing 1. Classification rules XML document
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE Classification SYSTEM "Classification.dtd" > <Classification schema_version="1.0"> <!-- HTTP Classification Rules --> <InboundClassification type="http" schema_version="1.0" default_transaction_class="TC_1"> <http_classification_info transaction_class="TC_2" uri="/MigrateHelloWorldWAR/*" description = "HW1" /> <http_classification_info transaction_class="TC_3" uri="/MigrateHelloWorldWAR2/*" description="HW2" /> </InboundClassification> <!-- IIOP Classification Rules --> <InboundClassification type="iiop" schema_version="1.0" default_transaction_class="TC_4"> <iiop_classification_info transaction_class="TC_5" application_name="TestEJBEAR" module_name="TestEJB.jar" description="Sample EJB Classification" /> </InboundClassification> <!-- MDB Classification Rules --> <SibClassification type="jmsra" schema_version="1.0" default_transaction_class="TC_8"> <sib_classification_info transaction_class="TC_9" bus="BUS1" destination="ASSURE1" description="SIB MDB" /> </SibClassification> </Classification>
As you can see, this document contains three sections, one for each classification rule type: http for web applications, iiop for EJB applications, and jmsra for SIB MDB based applications. The default_transaction_class fields are assigned values that will be used as defaults for that type of work. For more specific HTTP classification rules, the uri field holds the context root of the deployed WAR. For example, the value:
ensures that the transaction class TC_2 will be invoked whenever that context root is used, such as:
With IIOP (EJB) classification rules, the application (EAR) name and module name (JAR) must be provided.
With SIB MDB applications, the SIB name and the destination that the MDB is “listening” to is provided. Typically, an individual MDB will listen to its own destination. This means that an individual application's CPU usage can be readily determined.
The full Classification.dtd associated with the XML is shown in Listing 2.
Listing 2. Classification rules DTD document
<?xml version='1.0' encoding="UTF-8"?> <!ELEMENT Classification (InboundClassification|SibClassification)+> <!ATTLIST Classification schema_version CDATA #REQUIRED> <!ELEMENT InboundClassification ((iiop_classification_info* |http_classification_info*|endpoint*))> <!ATTLIST InboundClassification type (iiop|mdb|http) #REQUIRED> <!ATTLIST InboundClassification default_transaction_class CDATA #REQUIRED> <!ATTLIST InboundClassification schema_version CDATA #REQUIRED> <!ELEMENT iiop_classification_info (iiop_classification_info*)> <!ATTLIST iiop_classification_info activity_workload_classification CDATA #IMPLIED> <!ATTLIST iiop_classification_info application_name CDATA #IMPLIED> <!ATTLIST iiop_classification_info component_name CDATA #IMPLIED> <!ATTLIST iiop_classification_info description CDATA #IMPLIED> <!ATTLIST iiop_classification_info method_name CDATA #IMPLIED> <!ATTLIST iiop_classification_info module_name CDATA #IMPLIED> <!ATTLIST iiop_classification_info transaction_class CDATA #REQUIRED> <!ELEMENT endpoint (classificationentry*)> <!ATTLIST endpoint defaultclassification CDATA #REQUIRED> <!ATTLIST endpoint name CDATA #REQUIRED> <!ATTLIST endpoint type (messagelistenerport) #REQUIRED> <!ATTLIST endpoint description CDATA #IMPLIED> <!ELEMENT classificationentry EMPTY> <!ATTLIST classificationentry classification CDATA #REQUIRED> <!ATTLIST classificationentry selector CDATA #REQUIRED> <!ATTLIST classificationentry description CDATA #IMPLIED> <!ELEMENT http_classification_info (http_classification_info*)> <!ATTLIST http_classification_info host CDATA #IMPLIED> <!ATTLIST http_classification_info port CDATA #IMPLIED> <!ATTLIST http_classification_info uri CDATA #IMPLIED> <!ATTLIST http_classification_info description CDATA #IMPLIED> <!ATTLIST http_classification_info transaction_class CDATA #REQUIRED> <!ELEMENT SibClassification (sib_classification_info+)> <!ATTLIST SibClassification type (jmsra|destinationmediation) #REQUIRED> <!ATTLIST SibClassification default_transaction_class CDATA #REQUIRED> <!ATTLIST SibClassification schema_version CDATA #REQUIRED> <!ELEMENT sib_classification_info EMPTY> <!ATTLIST sib_classification_info transaction_class CDATA #REQUIRED> <!ATTLIST sib_classification_info selector CDATA #IMPLIED> <!ATTLIST sib_classification_info bus CDATA #IMPLIED> <!ATTLIST sib_classification_info destination CDATA #IMPLIED> <!ATTLIST sib_classification_info discriminator CDATA #IMPLIED> <!ATTLIST sib_classification_info description CDATA #IMPLIED>
It is convenient to create a directory called /classification in the relevant application server profile on the HFS file system. Both the DTD and the XML are uploaded (for example, via ftp in binary) to this location. (See Resources for more information.)
A WebSphere Application Server environment variable called wlm_classification_file is defined to reference the classification rules XML document. This variable can be set at the cell, node, or server level. If the cell or node level is specified, the information contained in it must be accessible and applicable to all servers that inherit the specification from that cell or node.
In the administrative console navigate to Environment > WebSphere variables > New. The Name of the new variable is defined as wlm_classification_file and the Value similar to (for example) /WebSphere/Base/AppServer/profiles/default/classification/zwlm.xml (Figure 1).
Figure 1. Setting the wlm_classification_file environment variable
Creating WLM classification rules
From the XML definition (Listing 1), you could see that seven transaction classes are specified, arbitrarily named: TC_1, TC_2, TC_3, TC_4, TC_5, TC_8, and TC_9. Seven WLM classification rules need to be created to utilize these classes. To do this:
- Start the z/OS WLM application according to the the installation instructions (typically the iwmarin0 command is executed).
- On the Choose Service Definition screen, select the Extract
definition from WLM couple data set (Figure 2).
Figure 2. Extract service definition for defining classification rules
- Select the Classification Rules option (Figure 3).
Figure 3. Classification rules are then selected using option 6
If there already exists a CB type service class for WebSphere Application Server classification, as shown in Figure 4, then this can simply be modified by selecting option 3 against it.
Figure 4. Select option 3 to modify the existing WebSphere classification service class
If a type CB classification rules service class for WebSphere Application Server does not exist, then it will be necessary to create one using option 1. Some examples of creating these classes for WebSphere Application Server are available in Defining Component Broker (CB) Classification Rules.
- When this WebSphere Application Server classification service class has
been selected, classification rules are inserted. In Figures 5 and 5a, option I,
Insert rule, is selected repeatedly to enter each classification rule. Each transaction class specified above in Listing 1 should have its own classification rule. The transaction rule name corresponds exactly to the Qualifier name of each rule. Each rule should have its own report class with a name, while arbitrary, should have some affinity with the transaction classes in the rules.
For example, the report class and transaction class could share the same numerical suffix. In this test, the report classes are defined as KGREP1, KGREP2, and so on. The Type of the rule is TC (Transaction Class), and the Service Class field is left blank to be inherited from the (in this case) CBCLASS.
Figure 5. Defining classification rules for each transaction class
Figure 5a. Defining classification rules for each transaction class, continued
- Notice that TC_6, TC_7 and TC_10 in Figures 5 and 6 are not actually used
as part of this test. WCL* is a pre-existing rule that is also not used
during this test. When ENTER is pressed for the above screen, the user is
prompted to create the report classes specified. This is accepted with
another ENTER as shown in Figures 6 and 6a below.
Figure 6. Creating the report classes
Figure 6a. Creating the report classes, continued
- If ENTER is pressed, followed by the function key F3 (to exit), confirmation is given that the type CB subsystem has been successfully modified (Figure 7).
Figure 7. Successfully exiting the creation of the classification rules
- It is now necessary to install and activate the WLM definition that has been altered. Press the F3 function key to exit the Subsystem Type Selection List, and the console returns to the WLM definition menu.
- Place the cursor over the Utilities menu bar, and press ENTER (Figure 8).
- Select option 1, Install definition and press ENTER.
Figure 8. Install the changes made to the configuration
The message Service definition was installed. (IWMAM038) should now display.
- The service policy now needs to be activated so that it will be applied. To
achieve this, place the cursor over the Utilities menu and press ENTER.
this time, however, select option 3, Activate service policy.
Select the policy (Figure 9) and press ENTER.
Figure 9. Activating the service policy
Figure 9 shows there is only one service policy in this example environment and it is selected for activation. The message Service policy NORMAL was activated. (IWMAM060) should now display.
Creating the post-processing JCL
In order to post-process the SMF data collected during execution, some JCL is required to perform two functions:
- Dump the SMF records. These will contain the RMF report data (including CPU usage) that is being collected about the WebSphere Application Server applications.
- Generate the RMF report based on this data dump.
The RMF report can then be collected and analyzed for WebSphere Application Server application CPU usage.
The JCL used to obtain the RMF reports is shown in Listing 3 .
Listing 3. JCL to obtain RMF reports
//RMFRPT JOB MSGCLASS=H,MSGLEVEL=(1,1),REGION=0M /*JOBPARM SYSAFF=MV60 //SMFDUMP EXEC PGM=IFASMFDP //DUMPINA DD DSN=SYS1.MV60.MANC,DISP=SHR,AMP=('BUFSP=65536') //DUMPINB DD DSN=SYS1.MV60.MAND,DISP=SHR,AMP=('BUFSP=65536') //DUMPOUT DD DSN=&&SMF89,DISP=(NEW,PASS),SPACE=(CYL,(10,10)) //SYSPRINT DD SYSOUT=* //SYSIN DD * INDD(DUMPINA,OPTIONS(DUMP)) INDD(DUMPINB,OPTIONS(DUMP)) OUTDD(DUMPOUT,TYPE(30,70,71,72,73,74,75,76,77,78,79)) START(1600) END(1800) /* //RMFDTL EXEC PGM=ERBRMFPP,REGION=0M //SYSPRINT DD SYSOUT=* //MFPINPUT DD DSN=*.SMFDUMP.DUMPOUT,DISP=(SHR,PASS) //SYSIN DD * REPORTS(CPU,HTTP,IOQ,OMVS,PAGESP,PAGING,VSTOR(D)) REPORTS(DEVICE(DASD,COMM),XCF,CHANNEL,HFS) REPORTS(PAGING) REPORTS(PAGESP) REPORTS(CACHE(SUBSYS)) SYSRPTS(WLMGL(SCLASS,SCPER,RCLASS,RCPER,POLICY)) SUMMARY(INT,TOT) /* //
Notice the period during which these reports will be dumped and generated is defined in the START and END parameters (between 16:00 and 18:00 hours in this sample). The report Interval is 30 minutes by default, so reports are generated for 30 minute periods. This parameter can be changed to modify the default report interval by using the INTVAL parameter in the SMFPRM member of the SYS1.PARMLIB dataset. (See the Information Center for more information.)
Obtaining the performance statistics
When the WebSphere Application Server server is restarted after the XML is uploaded there should be evidence of this file being successfully recognized by WebSphere Application Server in the z/OS job logs, specifically in the control region logs. You should see a message similar to this:
The /WebSphere/V80S01S1/AppServer/classification/zwlm.xml workload classification file was loaded at 2 ...
If there is a problem with the XML (for example, if it is structured incorrectly or contains typographical errors) then a message similar to this will be seen in the logs, indicating that the XML file should be analyzed:
Problems parsing WLM classification XML file ...
To obtain the performance statistics:
- In the dataset member RMFRPT, the START and END variables are changed to encompass the period required. In this example, START and END were changed to 1000 and 1300 respectively. No submission of the JCL or a WebSphere Application Server restart is required for this.
- To obtain some statistical information for this test, the WebSphere Application Server applications were subjected to varying workloads between 11:10 and 11:25.
- At 11:35 the JCL was submitted as shown in Figure 10. Because reports are generated (by default) for half hour periods from hh:00 to hh:30 and hh:30 to hh:00, submitting after 11:30 ensures that all relevant statistics are captured from 11:00 to 11:30.
Figure 10. Submitting the JCL
As a result of this, the report job RMFRPT is SDSF can be seen on the PRINT queue by issuing the ST command.
Figure 11. RMF report on the z/OS print queue
Expanding RMFRPT, as shown in Figure 11, displays all the subcomponents of this job, shown in Figure 12.
Figure 12. Subcomponents of the RMFRPT job on the print queue
PPXSRPTS contains the workload activity report that is needed and can be downloaded for viewing in the usual manner. The other subcomponents contain other RMF reports that may also be of interest, but are outside the scope of this article.
Viewing the statistics
PPXSRPTS is downloaded into a file called REPORT.txt (Figure 13). This document contains workload activity for half-hourly segments for the period defined in your JCL (10:00 onwards in this example). The relevant section here is from 11:00 to 11:30 because that is when workload occurred with the WebSphere Application Server applications. This workload was heavily biased towards the web applications and these were required to do far more mathematical calculations (as a simple mechanism to consume CPU cycles) than the MDB and EJB based ones.
Figure 13. Application workload activity (REPORT.txt)
From Listing 1 and Figure 5 it is known that KGREP2 and KGREP3 are the report classes associated with the two web applications; KGREP5 is associated with the EJB based application and KGREP9 with the MDB based application.
The report shows a range of data that can be used to understand the performance characteristics of each application. Some of the most useful are:
- ENDED: Number of transactions that completed within the period.
- ENDED/S: Average transaction rate (ended per second) for this period.
- ACTUAL: Average elapsed time for each transaction.
- EXECUTION: Average elapsed time spent executing the transaction.
- QUEUED: Average elapsed time spent on WLM queue waiting to be executed.
- SERVICE TIME – CPU: Time in seconds consumed on all processors.
- SERVICE TIME – AAP: zAAP service time in seconds.
- SERVICE TIME – IIP: zIIP service time in seconds.
- APPL% - CP: % of CPU time used by transactions running on general purpose processors.
- APPL% - AAPCP: Percentage of CPU time used by zAAP eligible transactions running on general purpose processors (standard CPs). This is a subset of APPL% CP.
- APPL% - IIPCP: Percentage of CPU time used by zIIP eligible transactions running on general purpose processors. This is a subset of APPL% CP.
- APPL% - AAP: Percentage of CPU time used by transactions executed on zAAPs in the service or report class period.
- APPL% - IIP: Percentage of CPU time used by transactions executed on zIIPs in the service or report class period.
See the Information Center for information on other fields in the report.
Using the SERVICE TIME – CPU and APPL% - CP fields, the reports in Figure 13 show the larger amounts of CPU consumed by the web applications (KGREP2 and KGREP3). In a real world scenario where general CPU usage was deemed to be excessive, this technique could pinpoint those applications as a potential cause.
This information can now be used to enable application developers to concentrate efforts on optimizing the applications that are consuming the most resources. Profiling tools such as the IBM Monitoring and Diagnostic Tools for Java - Health Center are available for analyzing problem applications in more detail; further information on the use of these tools is outside the scope of this article.
The CPU usage of individual applications running on IBM WebSphere Application Server for z/OS that utilize different Java EE technologies can be ascertained by using WLM classification rules with a classification rules XML document to define which applications are associated with the transaction classes. With these deployed, Service Management Facility (SMF) data can be collected, enabling the generation of Resource Management Facility (RMF) reports for view the CPU data. This mechanism, as described in this article, can be valuable in determining why z/OS resources are being saturated when a large number of applications co-exist in WebSphere Application Server on a sysplex.
- Classifying z/OS workload
- Using transaction classes to classify workload for WLM
- WebSphere Application Server for z/OS process model overview
- WebSphere Application Server Performance Tuning Toolkit
- z/OS workload management planning
- IBM Healthcenter
- IBM developerWorks WebSphere