What was delaying one of my jobs earlier today?
Laurie072183 1200008KBT Visits (8891)
In a previous blog entry, “You can get there from here”, the new OMEGAMON XE on z/OS version 5.3.0 Near Term History support was introduced. The navigation paths between all supported workspaces were explained. The scenario used in this blog entry focuses on one of those areas of navigation. It explains how to determine the reason for an address space experiencing delays, and the address space or spaces causing them, during a specific historical timeframe.
Why was my job delayed Jason?
Jason, in systems programming, received a call from Rosie, an application user, that transactions serviced by one of her production started tasks on the ZPETPLX2 Sysplex ran very slowly between 8:40am and 9:00am earlier in the morning. The STC was CONNRPT running on system Z2.
To establish the cause of the delay Jason decides to use OMEGAMON XE on z/OS e3270UI Near-Term History support to look at the address spaces experiencing delays during the reported slow-down.
From the initial Enterprise Summary workspace, KOBSTART, Jason uses the All Sysplexes sub-panel to navigate to the menu for Sysplex ZPETPLX2 by selecting the “H” option from the Options Menu.
Using the Historical Summary For CPCs Serving Sysplex workspace
Jason has previously set the Near-Term History summary time-frame to display the most recent 12 hours of data using history time configuration (View -> HIstory Time-span) so that the 8:40am to 9:00am time-frame is available to select from in the set summary interval.
To show execution delays for all address spaces in Sysplex ZPETPLX2 for a specific 5-minute interval Jason enters an “X” to the left of the recording date/time and presses ENTER to select Historical Sysplex Delay Details.
Using the Historical Sysplex Delay Details workspace
Workspace KM5WSCXH, Historical Sysplex Delay Details, provides a Sysplex-wide view of all delays by address space. Summary delay categories are displayed in each column for CPU, Enqueues, Devices, Storage, Subsystems (JES, HSM and XCF) and Operator requests. Additionally, address space Velocity and Total Using statistics are displayed. Five of the categories, Total, CPU, Device, Storage and Operator delays have zoom support to navigate to more detailed statistics in each of these categories simply by placing the cursor in one of these columns on an address space row and pressing ENTER.
Jason observes address space CONNRPT on system Z2 has a much lower than expected Velocity of 24% (Total Using / Total Delay + Total Using), an indication of being delayed for CPU and/or device I/O. He also observes that CONNRPT is experiencing a Total Delay of 72%, all of it attributable to CPU Delay during the 5-minute interval, in this case 8:35am to 8:40am.
Since this is CPU Delay Jason wants to focus on a single system, in this case Z2, so that he can see all of the other address spaces that may be impacting CONNRPT.
By selecting option “Z” from the Options Menu, Historical System Delay Details, Jason narrows down the Execution and Delays to Z2, the system that CONNRPT is running on.
Using the Historical System Delay Details workspace
The Historical System Delay Details essentially filters the Sysplex-wide execution and delays down to a single system, in this case system Z2.
Jason wants to see which address spaces may be impacting CONNRPT for CPU, but only on system Z2, so he places his cursor under the statistic in the Total Delay Percentage column and presses ENTER to zoom navigate to the Historical Address Space Delay Details workspace
Using the Historical Address Space Delay Details workspace
The Historical address Space Delay Details workspace consists of 5 sub-panels, one with summary delays and the others 4 showing detailed statistics for CPU, Device I/O, Storage and Operator delays.
Jason observes that the three top impactors on CONNRPT being able to access standard CP resource are FLASHSCM, *ENCLAVE (enclaves in general) and CICS3A2A, each impacting CONNRPT over 50% of the time it attempts to run on a standard CP over the 5 minute interval.
One of the impactors, FLASHSCM, catches his attention because he knows it is a test job that another application group has been running lately, but only during off-peak periods in the early morning hours. He uses the navigation buttons at the bottom of the workspace to scan the previous 5-minute intervals and observes that FLASHSCM is consistently impacting CONNRPT, sometimes by up to 95%. He informs the group that owns the FLASHSCM started task and asks them not to run it outside of their assigned testing hours in the future.
This blog entry has described some of the Execution and Delay Near-Term History support added in OMEGAMON XE on z/OS Version 530.
For additional Near Term History support and other features the capabilities described above are available with the OMEGAMON Performance Management Suite!