Why was work running slowly? Find out with OMEGAMON XE on z/OS
LaurenceGHart 100000EQEJ Visits (10415)
In a previous blog entry, “You can get there from here”, the new OMEGAMON XE on z/OS version 5.3.0 Near Term History support was introduced. The navigation paths between all of the supported workspaces were explained. The scenario used in this blog entry focuses on one of those areas of navigation. It explains how to determine which address space, or address spaces, might have been impacting the performance of a workload running on an LPAR during a specific historical time-frame.
Margrit we have a problem!
Margrit, a systems programmer, receives a call from operations that some of her application users have reported a variety of response time problems and batch delays earlier in the day. These occurred on a production Sysplex, ZPETPLX2, specifically one LPAR, Z2, in the 1:00pm to 2:00pm timeframe. This report is fairly general, but does suggest a CPU resource problem, so she decides to investigate starting at the CPC level where LPAR Z2 is running by selecting option “H” from the Enterprise Summary sub-panel of an e3270 session’s main KOBSTART workspace. This navigates to the workspace KM5CPC1H, Historical Summary For CPCs Serving Sysplex.
It’s worth mentioning that Margrit first sets the near-term historical time-frame for navigation as 1:00pm to 2:00pm today by selecting the View pull-down then option “I – History Timespan” . In the resulting KOBHIST “History Selection” pop-up, superimposed on the screen-shot below, she selects option 3 “Historical Time range” and sets the start and end times. By pressing the ENTER key, the time range for Near Term History navigation is set.
Using the Historical Summary For CPCs Serving Sysplex workspace
Workspace KM5CPC1H, Historical Summary For CPCs Serving Sysplex, provides an overall view of the CPCs that the LPARs in a Sysplex are running on. The total physical utilization percentages of the standard and specialty CPs are displayed, along with the effective service delivery rate and cycle steering information. The statistics represent all 5-minute periods in the configured historical time range for each CPC. In this scenario Margrit knows that the LPAR experiencing problems runs on CPC 0FBAA6. Reviewing the physical utilization of the standard CPs across the CPC over the 1 hour time-frame selected Margrit sees no complex-wide issues with the values reported. Also, because the Capacity Indicator values are 100, it’s clear that no cycle steering due to operator control or hardware malfunction has occurred that would reduce the level of service delivered to the CPC.
Margrit decides to investigate more detailed information for CPC 0FBAA6 by selecting one of the the 5-minute intervals of the 1-hour reporting time-frame (the default navigation selection). This action navigates to the KM5CPCDH workspace, Historical Details For A CPC.
Using the Historical Details For A CPC workspace
In the KM5CPCDH workspace Margrit observes that LPAR Z2’s Total Logical standard CP is around the 99% level for the 13:40pm to 13:45pm 5 minute interval. This is well beyond the normal utilization on Z2 at this time of day. Margrit could scroll chronologically back and forth in the KM5CPCDH workspace to investigate all of the statistics related to LPAR Z2 within the reporting time-period but takes the default navigation for LPAR Z2 so that a more convenient view of just this LPAR’s statistics is displayed.
By selecting LPAR Z2 and navigating to the Historical Details For An LPAR workspace, KM5LPRDH,Margrit can now focus on the statistics for the LPAR of interest and scroll chronologically back and forth in the reporting time-frame.
Navigating chronologically backwards and forwards in Historical Details For An LPAR using the navigation buttons at the bottom of the workspace Margrit observes the Total Logical standard CP around the 99% level for most of the intervals between 1:00pm and 2:00pm. This is well beyond the normal utilization on Z2 at this time of day.
Note that investigating a specific LPAR within the reporting time-frame using the KM5LPRDH workspace is more convenient than looking at LPAR statistics in the KM5CPCDH workspace because all statistics for each configured processor pool type are presented in a format that doesn’t require using left and right scrolling to look at statistic displayed in the current view.
Where is LPAR Z2’s CPU resource being used?
Now that Margrit has established that LPAR Z2 Total Logical CPU Utilization during the reporting time-frame is much higher than normal she wants to determine what address space or address spaces may be consuming more CPU resource than they should be.
Since the 1:00pm to 2:00pm reporting timeframe falls in the most recent 4 hours Margrit can take advantage of new navigation added to the existing KM5MSUO workspace, 4-Hour Rolling Average MSU Statistics. To navigate to this workspace from the Historical Details For An LPAR workspace she can use fast-path navigation by entering =KM5MSUO on the workspace command line and pressing ENTER.
Once in the KM5MSUO workspace, within the 4 hour range of intervals available Margrit can see the 5-minute time intervals in the 1:00pm to 2:00pm reporting. The new navigation added in this release is the “P” navigation character that navigates to the Interval CPU Utilization By Address Space workspace, KM5ASP1H, as shown below for the 1:30pm to 1:35pm time-frame.
Selecting each interval from KM5MSUO in the 1:00pm to 2:00pm reporting time-frame Margrit notices that one address space, FLASHSCM, is consistently using between 500% and 600% standard CPU. Margrit knows that FLASHSCM is a stress testing application that should only be run during off-peak hours. She informs the application owner so that appropriate action can be taken to free up the CPU resources for the suffering production work. The problem has been identified and resolved.
This blog entry has described some of the CPC and LPAR Near-Term History support added in OMEGAMON XE on z/OS Version 530.
For additional Near Term History support and other features the capabilities described above are available with the OMEGAMON Performance Management Suite!