Host configuration: The Fault Analyzer feature for the host is included as part of the IBM® Rational® Developer for System z™ Version 7.1 and later IBM® SMP/E for z/OS™ installation. In addition to the Rational Developer for System z host configuration, Fault Analyzer for z/OS V7.1 with PTF UK25564 or later must be configured in your environment. Consult the Fault Analyzer User's Guide and Reference for further information (SC19-1253).
for a link to the Rational Developer for System z V7.1 Host Installation Guide for full instructions.)
Client configuration: Fault Analyzer Integration (FAI) is an optional feature that you can choose to install by using IBM Installation Manager. This feature is listed in the Fault Analyzer User's Guide and Reference under "Integration with IBM Application Development and Problem Determination Tools."
The Fault Analyzer Integration feature is also available as a technology preview for IBM® WebSphere® Developer for System z, Version 7.0 (see Resources for a download link). In addition to installing and configuring it on the host machine, that version requires extensions for the host machine that are available in SMP/E format (software management program extended). Also, Fault Analyzer for z/OS V7.1 with PTF UK23264 must be installed. The client extension can be installed by using the Installation Manager.
Introduction to this feature
IBM® Fault Analyzer for z/OS™ is part of the z/OS Problem Determination and Deployment Tools, which have been around for several years. The Fault Analyzer helps you identify the cause, analyze the failure, and fix the problem when one of your applications ends abnormally (an abnormal ending is commonly called an abend). It can analyze abends of batch, IBM® CICS, IBM® Information Management System™ (IMS™), IBM® DB2® database, IBM® WebSphere®, and IBM® WebSphere® MQ™ applications.
The Fault Analyzer supports several different analysis modes:
- Real-time abend analysis: When a program abends (ends abnormally), the abend processing is intercepted and the Fault Analyzer is automatically invoked through an exit appropriate for the processing environment. The Fault Analyzer analysis records details about the abend in a fault history file (either PDS, partitioned data set, or PDSE, partitioned data set extended). A fault entry (a member in a fault history file) contains the result of abending program analysis.
The Fault Analyzer feature is invoked to analyze the abending program through LE CEECXTAN, LE CEEEXTAN, CICS XPCABND, and MVS IEAVTABX exits. See the Fault Analyzer User's Guide and Reference for further information (SC19-1253).
- Real-time SNAP analysis: The Fault Analyzer performs the same analysis as the real-time abend analysis, except that the SNAP analysis interface is explicitly invoked by your application where desired.
- Batch reanalysis: The Fault Analyzer performs the same analysis as the real-time abend analysis, except that it gets the required information from the saved information in a fault entry. You can specify additional information to get a more detailed analysis of the abending program. For example, you can make a listing (or a side file) available if one was not available during the real-time analysis. The Fault Analyzer obtains the required information from the saved minidump (or SYSMDUMP, or both) instead of the abending program's virtual storage.
- Interactive reanalysis: The interactive reanalysis presents information about an abending program as a series of ISPF (Interactive System Productivity Facility) panels. This enables you to choose the sections of interest. You can also view the contents of storage areas that are not necessarily formatted in the report generated by other methods. This provides additional information for certain types of abends (for example, IBM® CICS® and Java™ technology).
Fault history files are partitioned data set extended (PDSE) data sets that contain information about faults that have been analyzed by the Fault Analyzer. Fault entries, which are stored as separate members in the fault history file, contain the following types of information:
- Real-time analysis key information, such as abend code and failing program name
- Execution environment details, such as job name, system ID, and the date and time that the fault occurred
- Associated real-time analysis report (if applicable)
- Saved minidump pages (minidump pages are virtual storage pages that are accessed during the analysis, which are stored as part of a fault entry)
- Name of associated SYSMDUMP or SVC dump data set (if applicable)
The Fault Analyzer Integration (FAI) feature of Rational Application Developer for System z, V7.1 and later, enables you to work with history files and fault entries. This helps you identify the cause of failure in your applications.
How it differs from the Debug Tool for IBM z/OS
Rational Developer for System z V7.1 helps make the development of your traditional mainframe applications, Web applications, Web services, and XML-based interfaces faster and easier. In addition, IBM offers several problem determination tools (PD tools), including the Debug Tool for z/OS, File Manager for z/OS, and Application Performance Analyzer for z/OS, just to name a few. Both Rational Developer for System z V7.1 and PD tools can improve the efficiency of the development team, as well as the quality of the end applications.
A development tool such as Rational Developer for System z goes hand in hand with a debug tool while developing an application. A debug tool is a must-have tool to step through your application while examining its status, particularly during the development cycle (as opposed to deployment or production). The Fault Analyzer feature, by comparison, helps a slightly different user by filling the need as a problem determination tool in a near-production environment when an unexpected failure occurs in the application. Based on the status of the abending application (real-time analysis) or the analysis of system dump captured at the time of failure, the Fault Analyzer produces a report that describes the cause of failure. There are three key differences between the Fault Analyzer and the Debug Tool for z/OS:
- The Fault Analyzer does not require additional resources while your application is working normally. It is invoked only when an abend occurs.
- When you examine an application, you have a good idea where the problem exists. The knowledge of problematic areas enables you set break points in the application and examine the status of the live application. The Fault Analyzer is invoked when it is needed (when an abend occurs, for instance). You are not expected to specify the potential problematic areas.
- The Debug Tool enables you to examine the status of your live application as you go through the code. Fault Analyzer reports the status of your application at the time of failure, which gives you a snapshot of what was happening in your application.
Key features of Fault Analyzer Integration for Rational Developer for System z V7.1
The Fault Analyzer plug-in is the newest addition to the suite of Rational Developer for System z tools that help you control every part of the software design and deployment process. The plug-in simplifies abend analysis by connecting to a remote IBM z/OS system and displaying the result of the analysis in the familiar workspace.
Fault Analyzer Artifacts view
The Fault Analyzer Artifacts view (Figure 1) enables you to register a set of history files to browse fault entries to identify the cause of failures in your applications, and to monitor the contents of history files to alert you when a failure occurs in one of your applications.
History files are organized in a tree structure in which the root of the tree is FA Artifacts; which contains children that represent different mainframe systems. Each system node potentially contains two children, one for containing all history files for the system and another for containing the view information. When you double click on a history file element, all fault entries contained in the selected history file are displayed in the Detailed View (page 5). A view is a Fault Analyzer concept that enables you to group a set of history files. When you select a view element, fault entries contained in all history files defined in the selected view are displayed in the Detailed View.
A view enables you to group a set of history files, as well as to define the column layout for the Fault Entry List display in the ISPF environment. However, the column layout information defined in the view definition is ignored by the Fault Analyzer Integration feature at this stage.
Views must be defined on the host and retrieved using Retrieve View Information action from the context menu in the Fault Analyzer Artifacts view (FA Artifacts). See the "Setting up views" section in the Fault Analyzer User's Guide and Reference (SC19-1253).
Figure 1. FA Artifacts view
When you invoke the Fault Analyzer ISPF interface, a set of history files that you are using is stored as part of your ISPF profile information. To retrieve the information of your last-accessed history files:
- In the Artifacts view, select Retrieve Last Access History Files.
- When theRetrieve ISPF Profile dialog displays, use the arrow for the
drop-down menu to select the following information for these fields (also see
- Existing RSE Connection: PTHAPC1
- ISPF Profile Name: USERID().APC1ISPF.ISPPROF
- Temporary DSN: USERID().FA.COPY.ISPF.ISPPROF
- Exec Name: IDIC1.V7R1MO.SIDIEXEC(IDICSXDS)
- FA Application ID: IDA
Figure 2. Retrieve ISPF Profile dialog
Table 1 describes these fields.
Table 1. Field names and descriptions
|Existing RSE Connection||Name of the RSE (remote system explorer) connection profile. The ISPF profile information is retrieved from the specified host.|
|ISPF Profile Name||Name of a data set that contains your ISPF profile.|
|Temporary DSN||This is the name of temporary data set that is used to copy your ISPF profile. You must have permission to allocate and write to the data set.|
|Exec Name||Data set name that contains REXX (Restructured Extended Executor, or
|FA Application ID||Application ID of Fault Analyzer installation on the host. The default value is IDI (for initial domain identifier).|
Table 2 is a summary of all actions available in the FA Artifacts view.
Table 2. Summary of FA Artifacts view actions
|Add new history file||Enables you to add a history file from a particular host to the view.|
|Retrieve last accessed history file||Enables you to retrieve the set of history files you last used on the host, based on the information stored in your ISPF profile.|
|Retrieve view information||Enables you to retrieve information about views, which are stored in a data set.|
|Delete from view||Deletes the selected item from the view.|
|Populate/refresh history file||Refreshes the cached information about the selected history file.|
|Configure history file monitor||A history file monitor can be configured to monitor the changes made to a history file on the host. Whenever changes are made to the selected history file, the user is notified.|
|Set encoding||Enables you to specify the encoding for the selected system. That is, the language option used when the fault entry was created.|
The detailed view (Figure 3) displays the summary of fault entries contained in the selected history file or view in the FA Artifacts view. This view gives you a quick summary of what is happening on your system (history file-centric view of the system). The column headings are configurable, depending on your preference and area of your interest.
Figure 3. FA details view
From this view, you can invoke the Report browser (see the "Fault Analyzer Report browser" section). This will enable you to see the detailed analysis of the selected fault entry or to invoke the Dump browser (see the section that follows) to browse a hex dump display of minidump pages stored in the selected fault entry.
Table 3 shows a summary of all actions available from this view.
Table 3. Summary of detailed view actions
|Browse report||Retrieves the report associated with the selected fault entry and displays the report. The report is cached locally after it is retrieved.|
|Browse minidump||Retrieves the minidump pages associated with the selected fault entry and displays the pages. When the pages associated with a fault entry are retrieved, they are cached locally.|
|Delete cached data||Deletes the cached information for the selected fault entry.|
|Set encoding||Allows you to select an encoding for the system where the selected history file or view is located.|
Working with older fault entries
For a Fault Analyzer Integration client to show fault entries in a history file correctly, the fault entries must be created with the correct version of the Fault Analyzer feature, with the appropriate set of options specified. In these cases, a fault entry cannot be displayed correctly by using the Fault Analyzer Integration client:
- A fault entry was created by using an older version of the Fault Analyzer.
- A fault entry was created without the WDZClient option.
- A fault entry was created by using the DeferredReport option.
In such cases, the Fault Analyzer Integration client detects the invalidity of the selected fault entry and displays the Fault Entry Refresh dialog (Figure 4), so that you can refresh the contents of the fault entry by rerunning the analysis with the correct set of options.
Figure 4. Fault Entry Refresh dialog
Fault Analyzer Report browser
The Fault Analyzer Report browser (Figure 5) is a multi-tabbed browser that displays the report associated with the selected fault entry. The browser is typically invoked by running the Browse Report action in the detailed view.
Figure 5. Report browser
Different parts of a report are organized by using tabs for categories that show along the bottom of the screen. The tabs separate the report into these five categories and related subcategories:
- Main Report
- Fault summary section: A brief description of the fault, which includes the hyperlinked source line number of the abending program (if possible) and the abend code. If you click on the source line number, it displays the source code of the abending program where the failure occurred (see the "Source code display" section for further information). If you click on the abend code, it displays the description (see the "Lookup view" topic that follows).
- Synopsis section: The synopsis section provides a brief description of the fault and its analysis.
- Event Summary
- The Event Summary is a list of all events, in chronological order.
- The Event details subsection provides detailed information about each event. Additional information associated with the event, such as message description and the contents of the program's working storage, are included in this subsection. The source code information or failing machine instruction is also included here.
- Abend Job Information
- This section provides information about the abending job associated with the
real-time invocation of the Fault Analyzer feature, such as:
- abend date
- job ID
- job name
- job step name
- execution environment
- Language Environment runtime options
- This section provides information about the abending job associated with the real-time invocation of the Fault Analyzer feature, such as:
- Systemwide Information
- This section contains, for example, console messages that are not identified as belonging to any specific event or CICS system-related information, such as trace data and 3270 screen buffer contents. Information about open files that could not be associated with any specific event might also be included here.
- If there is no information in this section, then it does not appear in the report.
- Miscellaneous Information
- Fault Analyzer options: Lists of the Fault Analyzer options that were in effect at the time of the analysis.
- Prolog section: Information about the version, release, and modification level of the Fault Analyzer, as well as the latest APAR (authorized program analysis report) or PTF (program temporary fix) installed.
- Epilog section:Provides information about the invocation exists used and the approximate amount of above-the-line storage allocated during the analysis, followed by the fault ID assigned. It also includes the time and date when the report was created.
See the Fault Analyzer User's Guide and Reference (SC19-1253) for a comprehensive description.
Fault Analyzer Dump browser
Fault Analyzer Dump browser (Figure 6) is a multi-tabbed browser that enables you to browse the minidump pages stored in the selected fault entry in a history file. This browser is typically opened by selecting the Open Mini-Dump action against a fault entry in the summary view. Minidump pages are the virtual storage pages that are accessed during analysis and stored as part of a fault entry.
Figure 6. Minidump Hex Dump display
Minidump pages are annotated with the key information that the Fault Analyzer found during the analysis (dictionary data). This dictionary data includes information such as various control blocks, modules, Multiple Virtual Storage (MVS) areas, and register values. A summary of the dictionary data associated with the selected fault entry is presented in the Outline view (Figure 7) or in the Dictionary Data tab within the Dump browser.
Figure 7. Outline page for Dump browser
Source code display
The Fault Analyzer supports several different side-file formats to display the associated source information of an abending program. One of the supported formats is called the LANGX file, because it is produced by running IDILANGX utility program against a listing file produced by a compiler.
The Fault Analyzer Integration option supports the source line display of the abending program if the correct LANGX file was available during the analysis (see Figure 8). If the correct LANGX file was available during the analysis of your abending program, the source line number of the program is hyperlinked in the report. (The location of your LANGX file is specified in the IDILANGX DD card in your job step.) When you click the source line number (Line 17 in the example in Figure 8: PROCEDURE DIVISION), it retrieves the LANGX file from the host and generates the source line for display. The source file opens in your favorite editor in read-only mode, and the line where the abend occurred is highlighted.
Figure 8. Source line display
The location of LANGX files is specified in the IDILANGX DD statement in your job. Alternatively, you are given an opportunity to specify the data set names containing your LANGX files during the fault entry refresh process (Figure 9).
Figure 9. LANGX Data Set Name dialog
In the Lookup view, you can browse to the description about abend codes, messages, and other miscellaneous information. Figure 10 shows the description of IMS user abend U0002. You can browse the information in one of three ways:
- Find the description for a particular abend code, message, or miscellaneous information by navigating through the tree structure.
- Type in a pattern name in the Search box. All matching abend codes, messages, and miscellaneous information are displayed in the Results tab in the view. From the list of matching results, you can browse to find the description.
- An abend code associated with a particular fault entry is hyperlinked in the report view. Click the hyperlinked abend code to display the description.
Figure 10. Lookup view
You can switch between English or Japanese descriptions for abend codes, messages, and miscellaneous information.
- Select the Language Setting action from the drop-down menu within the view.
- When the Lookup Language Setting dialog displays, choose your preferred language.
The authors thank Rod Turner, Clive Nealon, and Oren Nachman for reviewing the early draft of this article.
|Column heading descriptions||FaultAnalyzerColumnHeadings.pdf||36KB|
- Read the Fault Analyzer User's Guide and Reference for more details.
- Consult the IBM Rational Developer for System z Host Configuration Guide (SC23-7658-00) for instructions on how to configure the Rational Developer for System z functions and servers on your z/OS host system.
- See the Rational Developer for System z product page for more information about how to improve the efficiency of traditional mainframe, Web development, and composite applications.
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- Subscribe to the developerWorks Rational zone newsletter. Keep up with developerWorks Rational content. Every other week, you'll receive updates on the latest technical resources and best practices for the Rational Software Delivery Platform.
- Browse the technology bookstore for books on these and other technical topics.
Get products and technologies
- Download the Technology Preview of WebSphere Developer for System z Fault Analyzer.
- Download trial versions of IBM Rational software.
- Join the Rational Developer for System z forum to get answers to your technical questions and participate in discussions.
Dig deeper into Rational software on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Keep up with the best and latest technical info to help you tackle your development challenges.
Software development in the cloud. Register today to create a project.
Evaluate IBM software and solutions, and transform challenges into opportunities.