System z Monitoring and Application Management

Home > Integration Scenarios > Using OMEGAMON XE for CICS with CICS Performance Analyzer for z/OS

Using OMEGAMON XE for CICS with CICS Performance Analyzer for z/OS

IBM® Tivoli® OMEGAMON XE® for CICS® on z/OS® is a complete monitoring solution that enables you to monitor and manage complex CICS systems by addressing potential problems quickly.

OMEGAMON XE for CICS is a component that uses the Tivoli Enterprise Portal as its user interface. The Tivoli Enterprise Portal is a highly customizable Java application that can be run as a desktop application or as an applet in your browser. Through this interface, you can monitor all your CICS regions and the other systems in your environment.

Figure 1 shows a typical Tivoli Enterprise Portal screen, with the navigation tree on the left expanded to show the monitored CICS regions, and no current alerts. You can also see from the open situation count that four situations have fired in the last 24 hours and they have all been addressed and are no longer current. Each of these situations has been acknowledged, addressed and closed by an OMEGAMON user already.

Figure 1 - TEP screen

IBM CICS Performance Analyzer for z/OS is a CICS system performance reporting and analysis solution. It provides system management and measurement reports on all aspects of CICS application performance. You can use this program for capacity planning, tuning, and help in identifying trends leading to performance problems.

Performance Analyzer generates reports by using data collected from SMF records created from multiple products, including CICS, CICS Transaction Gateway, OMEGAMON, IBM DB2® and IBM WebSphere® Message Queue Series.

Performance Analyzer data can be accessed from the CICS Explorer. By using the Performance Analyzer program, you can perform data analysis of reports launched through a powerful ISPF (green screen) interface. The CICS Explorer is an integration framework that allows access to CICS itself - think CEDA and CEMT in a GUI. You can also extend the CICS Explorer integration by using Eclipse features. An example of the value of this integration is where Performance Analyzer has been used to view summary transaction data to determine transactions that are spending a lot of time thread-switching because of the incorrect use of the concurrency attribute for some programs. Having determined this transaction time from the Performance Analyzer feature plug-in in the CICS Explorer, you can jump directly into the System Management portion of the CICS Explorer to view and modify the CICS Program Definition and then discard and reinstall the program. Because Performance Analyzer collects SMF data going back over a period of time, you can distinguish between trends and spikes by comparing transaction data from different time periods, and build up a powerful view of the operational characteristics of CICS regions and their related subsystems.

OMEGAMON for CICS and Performance Analyzer work together by first employing OMEGAMON for CICS to help identify and resolve immediate application performance problems quickly, ensuring that immediate impacts to service are minimized before more thorough analysis using Performance Analyzer to investigate and understand the problem - whether it's a trend, a spike, a configuration problem, or something about the way the application has been written or deployed. By having both immediate and actionable data through OMEGAMON, you can take fast corrective action to CICS resources and operational settings, while using Performance Analyzer to identify long-term changes.

The scenario described here is typical of the sort of problem that can significantly impact the performance of a typical CICS system. In this scenario, you have a deadlock between applications in two CICS regions accessing a VSAM data set.

As shown in Figure 2 - 'CICS Transaction Analysis workspace with alert showing', you can see the transaction analysis for a CICS region, when an alert fires (on a different CICS region), but you can see the alert as the red icon in the navigation tree. The alert is also visible if you do not have the tree expanded to display the CICS region with the problem.

Figure 2 - CICS Transaction Analysis workspace with alert showing

You can expand the CICS region in the tree with the problem to investigate further, and this is shown in Figure 3 - 'OMEGAMON VSAM RLS Analysis with alert'. Here you see that task 106 is holding GBURGES.TESTFILE.KSDS while task 81 in a different CICS region is waiting on that resource. You can note the resource, task numbers and the transaction ID ('LOCK') for later analysis.

Figure 3 - OMEGAMON VSAM RLS Analysis with alert

Your first concern is to resolve this situation, and to do this, you need to free up the lock being held by task 106 so that task 81 can resume.

Clicking on the situation takes you to a workspace, shown in Figure 4 - 'Situation resolution workspace', which gives us diagnostic information about the situation, expert advice to guide you, and a take action panel that allows you to deal with the task holding the lock.

Figure 4 - Situation resolution workspace

An appropriate action here is to 'kill' the transaction holding the lock, and then to investigate the application to see why it has been holding the resource for so long. Selecting the take action option opens a dialog box such as that shown in Figure 5 - 'Take Action kill transaction dialog'. Here you want to use the 'CEKL' transaction to kill the blocking task. Click OK to terminate the task and free the lock.

Figure 5 - Take Action kill transaction dialog

When the task has been killed, the alert clears, and you can flag the situation as resolved. Obviously, your transaction recovery procedures must be invoked due to the abnormal ending of the task.

Now that the immediate problem has been resolved quickly, you can investigate further to see why the problem occurred. For this, you turn to CICS PA.

You will now look at using CICS PA to delve deeper into the previous problem. You will try to assess the impact of the problem on your CICS systems and applications.

You start by creating a new report set for your situation, which extracts the particular information that you are interested in from the SMF data. From the report sets panel, shown in Figure 6 - 'CICS PA New report Set', enter New to create a new report set.

Figure 6 - CICS PA New report Set

The report set selection panel is displayed, as shown in Figure 7 - 'Report Set selection panel'.

Figure 7 - Report Set selection panel

Here, you've selected the Wait Analysis and Cross-System Work reports to view your transaction activity (because you saw in OMEGAMON XE for CICS that the situation crossed two CICS regions) and the File Usage Summary report to determine the actual file names used by our transactions, so that you can find which one caused the problem.

You can drive each of these three reports for the transaction ID 'LOCK' that you noted when you saw the problem in OMEGAMON. CICS PA generates JCL that will drive the creation of the reports. You can save and/or edit the JCL if you want, and you can view the output in SDSF (System Display and Search Facility), the ISPF Outlist option, or in the new CICS PA Explorer plug-in.

A highlighted version of the Cross-system Work Report is shown in Figure 8.

Figure 8 - Cross system work report

In this report, the transactions that you noted earlier when dealing with the situation in OMEGAMON are highlighted. Also note that there are several other transactions in the report have long response times, so you appear to have a bigger problem than the single occurrence that you resolved in OMEGAMON.

Next, take a look at a highlighted version of the wait analysis report in Figure 9.

Figure 9 - Wait Analysis report

From the summary section of the report, note that transaction ID 'LOCK' spent most of its time suspended by the CICS dispatcher. In the details section, you can see that the transaction has spent most of its suspend time in either an Interval Control wait or RLS File I/O wait. You can conclude that the application is not releasing its record.

The last of the three reports that were selected is the File Usage Summary report, shown, once again in a highlighted form, in Figure 10.

Figure 10 - File Usage Summary Report

Here, you can see that the only file accessed by the transaction with ID 'LOCK' was 'TESTSKDS'. You can further see that the accesses were GET requests (EXEC CICS READ), and, given what you have already seen, it is not surprising that both the average and maximum times for these requests are very high.

Armed with this information, you can prompt the application programmers to take a look at this particular application, and give them sufficient information to greatly aid their investigation and fixing of the problem in the application code.

In this example, OMEGAMON XE for CICS was used to spot and resolve a problem that can significantly impact application performance, and then CICS PA was used to investigate the causes of the problem and give the application programmers sufficient information to help them fix the application problem. In a more realistic CICS production environment, these tools clearly can work together to ensure system and application performance, not just in the simple example demonstrated here, but in many other scenarios.

About the Authors

This scenario was written by John Taylor, who is a Tivoli OMEGAMON XE for CICS Developer, and Joe Winchester, a CICS Tools Developer.