Measuring API workloads with WLM

As you prepare to deploy APIs to z/OS® Connect EE, you might consider how to measure the number of API requests, and monitor response times and CPU cost.

You can use z/OS Workload Management (WLM) to classify API requests and the Resource Management Facility (RMF) to measure z/OS Connect EE API workloads.

Using WLM to classify API requests

WLM uses classification rules to associate incoming requests with service classes which specify appropriate performance goals. z/OS Connect workloads are managed by two types of WLM classification:

STC type classification
z/OS Connect EE servers can be classified as STCs (started tasks) in order to manage all of the server activity that does not relate to specific transactions, for example, garbage collector activity and connection pool management.
Note: A z/OS Connect EE server can also be started as a submitted job, in which case you would use an equivalent JES classification.
CB type classification
Every request to a z/OS Connect EE server is dispatched on a USS (Unix System Services) thread and executes under a WLM enclave that is classified as a CB workload transaction.

Every request to a z/OS Connect EE server is dispatched on a USS (Unix System Services) thread and executes under a WLM enclave that is classified as a CB workload transaction.

The processor activity of z/OS Connect EE transactions is reported to WLM and RMF and so can be measured in RMF Workload Activity reports for service classes and report classes. RMF Monitor I provides long-term data collection for system workload and system resource utilization. It measures various areas of system activity over a long period of time, and writes the data into SMF record types 70 and 72. The RMF Post Processor is used to create reports from RMF Monitor I data

With the recent WLM enhancements you can now also tag z/OS Connect EE requests with MOBILE reporting attribute (see article ‘Limiting the impact of API workloads on MLC’).

A transaction class provides a granular approach for classifying API requests. For instance, you can classify requests for different API operations inside the same z/OS Connect EE server depending on the request URI.

A transaction class for z/OS Connect EE is enabled using the zosWlm-1.0 feature. After adding this feature, validate that the z/OS Connect EE server is authorized to use the feature by checking for the message ‘CWWKB0103I: Authorized service group ZOSWLM is available’ in the messages log. For more information on authorizing z/OS Connect EE to use WLM, refer to

https://www.ibm.com/support/knowledgecenter/en/SS7K4U_liberty/com.ibm.websphere.wlp.zseries.doc/ae/twlp_wlmclassification.html

Using a transaction class to classify API requests

Transaction classes are defined in the server.xml file. Figure 1 shows an example classification of 3 API operations for the sample catalog API. The catalog API is used to access the CICS® Catalog Manager sample application. For more information on the catalog API, refer to the z/OS Connect EE Getting Started Guide.

Figure 1. Transaction class definition in server.xml
<zosWorkloadManager collectionName="MOPZCET"/>
<wlmClassification>
  <httpClassification
          transactionClass="TCIC"
          method="GET"
          resource="/catalogManager/items"/>
  <httpClassification
          transactionClass="TCIS"
          method="GET"
          resource="/catalogManager/items/*"/>
  <httpClassification
          transactionClass="TCPO"
          method="POST"
          resource="/catalogManager/orders"/>
</wlmClassification>

In Figure 1 we assign transaction class TCIC to the inquire catalog API operation which is invoked using an HTTP GET request with URI /catalogManager/items. We also assign transaction classes to the inquire single and place order API operations.

The resource attribute of the httpClassification element can be specified using wildcards. One asterisk represents any number of characters, and is used as a wildcard for the string between two path separator (/) characters. Two consecutive asterisks represent any number of characters that include the path separator. Double asterisks cannot appear with other characters within a pair of path separators.

For example, for the API /catalogManager/items, that has the base path catalogManager and the path items, you can use one of the following resource definitions in the httpClassification element to obtain SMF data:
  • /catalogManager/items
  • /catalogManager/*
The following resource definition will not match the API and no SMF data will be captured.
  • /catalogManager/items/*
A further example for API /rainbow/red/yellow/blue, that has the base path rainbow and the path red/yellow/blue, you can use one of the following resource definitions on the httpClassification element to obtain SMF data:
  • /rainbow/**/*
  • /rainbow/**/blue
The following resource definitions will not match the API and no SMF data will be captured.
  • /rainbow/*
  • /rainbow/**

In the final case, the message CWWKB0164W: WLM CLASSIFication Configuration value for attribute resource uses incorrect wildcarding. is reported in the messages.log file.

For API requesters, the resource must be prefixed with /zosConnect/apiRequesters/ . The following resource definition in the httpClassification will obtain SMF data for all API requesters:
  • /zosConnect/apiRequesters/*

For more information about defining a transaction class refer to WLM classification for z/OS in the WebSphere Application Server Liberty documentation.

A collectionName can be provided with the zosWorkloadManager XML element. If not defined, the default value of the collectionName is the server name, the name that appears in the servers directory in Unix System Services. The collectionName of the z/OS Connect EE server is used for WLM classification, so it needs to match the CN name in WLM as shown in Figure 2. This implies that the collectionName value is limited to 8 characters and has to be in uppercase. If the server name does not comply with these two conditions, a custom collectionName – complying with the restrictions – has to be set with zosWorkloadManager in order for the API workload to be classified properly.

Figure 2 shows the corresponding WLM classification rules for classifying the catalog API requests.

Figure 2. API WLM classification rules.

 Subsystem Type . : CB          Fold qualifier names?   Y  (Y or N)      
 Description  . . . WebSphere requests
        
 Action codes:   A=After     C=Copy        M=Move     I=Insert rule
                 B=Before    D=Delete row  R=Repeat   IS=Insert Sub-rule
                                                              More ===> 
          -------Qualifier--------                 -------Class--------  
 Action   Type      Name     Start                  Service     Report
                                          DEFAULTS: CBENCLAV    RCBDEF   
  _____ 1 CN        MOPZCET    ___                  CBHI        RCBZCET 
  _____ 2   TC        TCIC       ___                CBHI        RCBTCIC 
  _____ 2   TC        TCIS       ___                CBHI        RCBTCIS 
  _____ 2   TC        TCPO       ___                CBHI        RCBTCPO        

Figure 2 defines specific report classes to monitor the different API operations, for example, transaction class TCIC for inquire catalog requests is assigned to the RCBTCIC report class. We will use this report class to report on the number of inquire catalog API operations, the response time and CPU consumed.

Classifying the z/OS Connect EE server

Figure 3 shows the WLM classification rule for classifying the z/OS Connect EE server.

Figure 3. z/OS Connect EE server WLM classification rule.

 Subsystem Type . : STC         Fold qualifier names?   Y  (Y or N)      
 Description  . . . Started Tasks classifications
        
 Action codes:   A=After     C=Copy        M=Move     I=Insert rule
                 B=Before    D=Delete row  R=Repeat   IS=Insert Sub-rule
                                                              More ===> 
          -------Qualifier--------                 -------Class--------  
 Action   Type      Name     Start                  Service     Report
                                          DEFAULTS: OPSDEF      RSTCDEF
  ____ 1  TN       MOPZCET   ___                    SYSSTC      RSTCZCET 

Figure 3 defines a specific report class RSTCZCET to monitor the z/OS Connect EE started task (STC) MOPZCET. We will use this report class to report on the server activity that does not get attributed to specific API requests.

Measuring the API

Figure 4 shows an extract from the RMF Workload Activity report for inquire catalog API requests (report class RCBTCIC).

Figure 4. RMF Workload Activity report for report class inquire catalog API requests.

 REPORT BY: POLICY=WLMPOL  REPORT CLASS=RCBTCIC
                           DESCRIPTION =CB RC for inquire catalog API 
 -TRANSACTIONS--  TRANS-TIME HHH.MM.SS.TTT   SERVICE  TIME        ---APPL %---   
 AVG        0.04  ACTUAL                 2   CPU     7.176        CP      1.20 
 MPL        0.04  EXECUTION              1   SRB     0.000        AAPCP   0.00
 ENDED     11900  QUEUED                 0   RCT     0.000        IIPCP   1.20 
 END/S     19.83  R/S AFFIN              0   IIT     0.000                
 £SWAPS        0  INELIGIBLE             0   HST     0.000        AAP      N/A    
 EXCTD         0  CONVERSION             0   AAP       N/A        IIP      N/A 
 AVG ENC    0.04  STD DEV                0   IIP       N/A

 TRANSACTION APPL% :   TOTAL  :  CP  1.20   AAP/IIP ON CP 1.20   AAP/IIP 
                       MOBILE :  CP  0.00   AAP/IIP ON CP 0.00   AAP/IIP  

Figure 4 provides us with the following useful reporting information:

  • An average of 19.83 inquire catalog API requests were executed per second during the recording interval (10 minutes)
  • A total of 7.176 seconds of CPU time was consumed by the inquire catalog API requests, which equates to 0.6ms per request
  • The total CPU consumed is equivalent to 1.20% of a CP
  • 100% of the CPU consumed is zIIP eligible.
    Note: The IIPCP value represents the percentage of a CP that could have run on a zIIP if a zIIP had been available on the system.
  • No mobile CPU is reported since we have not classified the inquire catalog API requests as being initiated by a mobile device

Figure 5 shows a similar extract from the RMF Workload Activity report for inquire single API requests (report class RCBTCIS).

Figure 5. RMF Workload Activity report for report class inquire single API requests.

 REPORT BY: POLICY=WLMPOL  REPORT CLASS=RCBTCIS
                           DESCRIPTION =CB RC for inquire single API  
 -TRANSACTIONS--  TRANS-TIME HHH.MM.SS.TTT   SERVICE  TIME        ---APPL %---   
 AVG        0.03  ACTUAL                 1   CPU     3.930        CP     0.65        
 MPL        0.03  EXECUTION              1   SRB     0.000        AAPCP   0.00 
 ENDED     11900  QUEUED                 0   RCT     0.000        IIPCP   0.65        
 END/S     19.83  R/S AFFIN              0   IIT     0.000        
 £SWAPS        0  INELIGIBLE             0   HST     0.000        AAP      N/A 
 EXCTD         0  CONVERSION             0   AAP       N/A        IIP      N/A 
 AVG ENC    0.03  STD DEV                0   IIP       N/A

 TRANSACTION APPL% :   TOTAL  :  CP   0.65   AAP/IIP ON CP   0.65   AAP/IIP N/A                      
                       MOBILE :  CP   0.00   AAP/IIP ON CP   0.00   AAP/IIP N/A  

Figure 5 shows that a total of 3.930 seconds of CPU time was consumed by the inquire single API requests, which equates to 0.33ms per request. Note that this is less than the average CPU cost of inquire catalog API requests because an inquire single request retrieves a single item from the catalog, whereas the inquire catalog request retrieves 15 items.

Reporting on the z/OS Connect EE server

Figure 6 shows an extract of an RMF Workload Activity report for report class RSTCZCET. This is the report class that we defined in Figure 3 for the z/OS Connect EE server.

Figure 6. RMF Workload Activity report for z/OS Connect EE server.

 REPORT BY: POLICY=WLMPOL  REPORT CLASS=RSTCZCET
                           DESCRIPTION =STC classification for MOPZCET
 -TRANSACTIONS--  TRANS-TIME HHH.MM.SS.TTT   SERVICE  TIME        ---APPL %---   
 AVG        1.00  ACTUAL                 0   CPU     5.158       CP       0.92
 MPL        1.00  EXECUTION              0   SRB     0.339       AAPCP    0.00 
 ENDED         0  QUEUED                 0   RCT     0.000       IIPCP    0.85
 END/S      0.00  R/S AFFIN              0   IIT     0.000
 £SWAPS        0  INELIGIBLE             0   HST     0.000       AAP       N/A
 EXCTD         0  CONVERSION             0   AAP       N/A       IIP       N/A
 AVG ENC    0.00  STD DEV                0   IIP       N/A   
 
 TRANSACTION APPL% :   TOTAL  :  CP   0.92   AAP/IIP ON CP   0.85   AAP/IIP  N/A
                       MOBILE :  CP   0.00   AAP/IIP ON CP   0.00   AAP/IIP  N/A 

Figure 6 shows that a total of 5.158 seconds of CPU time was consumed by the server during the reporting interval (10 minutes). The total CPU consumed by the server is equivalent to 0.92% of a CP, out of which 0.85% of the CP is zIIP eligible.

Measuring the CICS transaction

Each request to the catalog API results in the execution of a CICS transaction. We also use WLM to classify the CICS transaction and measure the transaction response time and CPU consumption.

Note: Starting with CICS TS V5.3, transactions can now be classified into service and report classes, which directly measure and report their CPU consumption. The same support is available with IMS V14.

Figure 7 shows the WLM classification rules for classifying the CICS Catalog Manager transactions.

Figure 7. CICS transaction classification rules.

 Subsystem Type . : CB          Fold qualifier names?   Y  (Y or N)
 Subsystem Type . : CICS        Fold qualifier names?   Y  (Y or
        N)      
 Description  . . . CICS Environment
 Action codes:   A=After     C=Copy        M=Move     I=Insert rule
                 B=Before    D=Delete row  R=Repeat   IS=Insert Sub-rule
                                                              More ===> 
          -------Qualifier--------                 -------Class--------  
 Action   Type      Name     Start                  Service     Report
                                          DEFAULTS: CBENCLAV    RCBDEF  
 _____  1 SI        CICSMOBT ___                    CICSHI      RCICMOBT
 _____  2   TN        MZIC     ___                  CICSHI      RCICMZIC
 _____  2   TN        MZIS     ___                  CICSHI      RCICMZIS 
 _____  2   TN        MZPO     ___                  CICSHI      RCICMZPO
    

Figure 7 defines specific report classes to monitor the different CICS transactions, for example, inquire catalog requests invoke CICS transaction MZIC which is classified in the RCICMZIC report class. We use this report class to report on the number of MZIC transactions, the response time and CPU consumed.

Figure 8. RMF Workload Activity report for CICS transaction MZIC.

 REPORT BY: POLICY=WLMPOL         REPORT CLASS=RCICMZIC
                                  DESCRIPTION =RC for CICS transaction MZIC
 -TRANSACTIONS--  TRANS-TIME HHH.MM.SS.TTT   SERVICE  TIME        ---APPL %---   
 AVG        0.00  ACTUAL                 0
 MPL        0.00  EXECUTION              0
 ENDED     11900  QUEUED                 0
 END/S     19.83  R/S AFFIN              0

 TRANSACTION APPL% :   TOTAL  :  CP   0.39   AAP/IIP ON CP   0.00   AAP/IIP  N/A
                       MOBILE :  CP   0.00   AAP/IIP ON CP   0.00   AAP/IIP N/A     
Figure 8 shows that an average of 19.83 MZIC transactions were executed per second, with an average response time of 0ms (due to rounding). It also shows that 0.39% of a CP was consumed by these transactions.
Note: In the same way that we use an STC WLM classification in Figure 6 for the z/OS Connect EE server, you can also use an STC classification to report on the CICS server activity.

Calculating the cost of an API requester call

SMF72 records capture the CPU usage for all requests within a server. These records can be used to understand the cost of an API requester call.

Configure Report Classes in WLM in TSO and add the following definitions in server.xml.

<featureManager>
    <feature>zosconnect:apiRequester-1.0</feature>
    <feature>zosconnect:zosConnectCommands-1.0</feature>
    <feature>zosconnect:zosWLM-1.0</feature>
</featureManager>

<zosWorkloadManager collectionName="Liberty"/>

<wlmClassification>
    <httpClassification transactionClass="TCAPIREQ" method="PUT" resource="/zosConnect/apiRequesters/*"/>
</wlmClassification>
Define a classification rule. For example,

    Qualifier   Qualifier       Starting     Service   Report
 #  type        name            position     Class     Class
 -  ---------   --------------  --------     -------   -------
 1  CN          LIBERTY
 2  . TC        . TCAPIREQ                             APIREQ

Define a WLM report class APIREQ for API Requester.

Summary

This article gives an example of how WLM and RMF can be used to measure the performance of z/OS Connect EE APIs. Many factors can influence the performance of APIs, including message lengths, security and back end connectivity.

More information

More information on using WLM to measure z/OS workloads can be found in the ITSO Redbooks publication Using IBM z/OS WLM to Measure Mobile and Other Workloads (REDP-5359-00).

More information on z/OS Connect EE performance can be found in the IBM z/OS Connect Enterprise Edition Performance Reports.