A fix is available
APAR status
Closed as new function.
Error description
New Function
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: z/OS File System (ZFS) 2.5 * * FMID(HZFS450) * * * * z/OS File System (ZFS) 3.1 * * FMID(HZFS510) * **************************************************************** * PROBLEM DESCRIPTION: Providing two new ZFS health check * * functions to assist in monitoring * * performance and providing detailed file * * system information for mounted ZFS * * file systems that have one or more * * exception states. * **************************************************************** * RECOMMENDATION: APPLY PTF * **************************************************************** APAR OA63911 is providing two new health checkers that can be used to assist in monitoring and improving cache performance (ZFS_CACHE_PERFORMANCE) and also with identifying zFS file systems with exception states (ZFS_EXCEPTIONS).
Problem conclusion
Temporary fix
Comments
The new function for APAR OA63911 is being delivered for both z/OS 2.5 and 3.1 system releases. z/OS 2.5 users should refer to the 3.1 online documentation as the z/OS 2.5 online versions are not being refreshed. All online documentation updates will be provided in the 3.1 links listed below. For complete details and reference, refer to the following chapters and sections in the z/OS 3.1.0 IBM Documentation for information regarding the new function. Link to the documentation: For the zFS File System Administration and File System Messages and Codes guides, refer to the following link: https://www.ibm.com/docs/en/zos/3.1.0?topic=zos-file-system-zfs For the IBM Health Checker for z/OS User's Guide, refer to the following link: https://www.ibm.com/docs/en/SSLTBW_3.1.0/pdf/e0zl100_v3r1.pdf The following updates have been made in the z/OS 3.1 File System Administration SC23-6887-XX Summary of changes for z/OS 3.1 (New) You can identify zFS file systems that contain v4 directories and also convert v4 directories to v5 with the ioeconv4 command. See "ioeconv4" on page 129. "List Detailed File System Information" on page 298 was also updated. (APAR OA63911, which applies to 3.1 and 2.5) The reset_hcpstats value is added to the MODIFY ZFS PROCESS command. See "MODIFY ZFS PROCESS" on page 110. (APAR OA63911, which applies to 3.1 and 2.5) A new utility has been added: ioeconv4 Purpose ioeconv4 is a utility or REXX exec that identifies zFS file systems that contain v4 directories and also converts v4 directories to v5. If errors occur during the conversion of v4 directories to v5, ioeconv4 displays the error and starts converting the next v4 directory. Depending on the type of the error, you can rerun this conversion tool after you fix the error. ioeconv4 can be run from the z/OS UNIX shell via /usr/sbin/ioeconv4 or from the TSO/E environment via IOE.SIOEEXEC(IOECONV4). Format ioeconv4 {check|convert|convertro} [path|aggr=aggrname] Options aggr=aggrname Specifies the aggregate name that will have all its v4 directories processed according to the chosen option. If this option and the path option are omitted, all zFS file systems are checked and converted if appropriate. The aggregate name is not case-sensitive and is always converted to uppercase. check Checks for and report on directories that are v4. convert Converts any directories that are v4 to v5 on zFS file systems that are mounted read/write. If the aggregate version is not 1.5, it is also converted to 1.5. convertro Same as the convert option for zFS file systems that are mounted read/write. For zFS file systems that are mounted read-only, the file systems are remounted to read/write, and then the v4 directories area are converted to v5. If the aggregate version is not 1.5, it is also converted to 1.5. path Specify the path name of the directory in a zFS file system. If this option and the aggr=aggrname option are omitted, all zFS file systems are checked and converted as needed. If path is specified, the containing file system of that path will have all its v4 directories processed according to the chosen option. The name of the containing file system is displayed in the program output. Usage notes for ioeconv4 1. If an aggregate has the converttov5 attribute assigned to it, accessing a v4 directory with the ioeconv4 check option can cause its conversion to a v5 directory. For more information, see "Converting an existing v4 directory to an extended (v5) directory" on page 25. 2. If an aggregate contains many directories (for example, hundreds of thousands or millions of directories), ioeconv4 might take a long time to complete. 3. If there are many file systems mounted or there is heavy file system activity, the check option for all mounted file systems might take a long time to process. 4. To show file systems that contain v4 directories, you can use the D4 flag when you specify the zfsadm fsinfo -selection option. The file system reported from zfsadm fsinfo can be supplied to ioeconv4 for conversion as appropriate. 5. The ZFS_EXCEPTIONS health check displays the file systems with v4 directories that were mounted during IPL. For more information, see ZFS_EXCEPTIONS in IBM Health Checker for z/OS User's Guide. Restrictions 1. Directories that are still opened by an application cannot be converted. This program reports any v4 directory that cannot be converted. Privilege required The issuer must be logged in as a root user (UID=0). The following updates have been made in the z/OS 3.1 File System Messages and Codes SC23-6885-xx Summary of message changes for z/OS 3.1 New The following messages are new. IOEZH0076I IOEZH0077I IOEZH0078I IOEZH0079I IOEZH0080I IOEZH0083I IOEZH0084I IOEZH0088I New message information: IOEZH0076I zFS does not have enough performance data to do an analysis. Explanation: zFS needs more time to collect more performance data for Health Checker to analyze. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Automation: N/A Routing code: N/A Descriptor code: N/A IOEZH0077I File cache performance is good. No further action is needed. Explanation: Based on the collected statistics, overall file cache hit ratios are good. You do not need to take any action. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Automation: N/A Routing code: N/A Descriptor code: N/A IOEZH0078I Meta cache performance is good. No further action is needed. Explanation: Based on the collected statistics, overall meta cache hit ratios are good. You do not need to take any action. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Automation: N/A Routing code: N/A Descriptor code: N/A IOEZH0079I File cache total significant intervals = number number% of collected intervals showed sub-optimal performance data. Explanation: File cache total significant intervals is the number of observed intervals where file cache request threshold are met. % of collected intervals showing suboptimal performance data is the ratio of the total collected intervals with sub-optimal file cache performance to the file cache total significant intervals. This message indicates file cache performance is not optimal. A report of the time periods with sub-optimal performance will be displayed after this message. No action is required. zFS considers file cache performance good when the file cache hit ratio, (the ratio of requests to the cache that find the desired data in the cache divided by the total requests to the cache), is above 90%. In many cases, active systems that read file data stored in zFS file systems will achieve this hit ratio since many UNIX applications read the same files multiple times and zFS aggressively performs read-ahead for files. However, there are some situations where the cache hit ratio might be less, such as system startup when applications might be reading large volumes of data for the first time, or during administration actions such as moving ownership of file systems and other zFS administration commands. If data that is not read frequently is accessed, a temporary dip in the hit ratio might occur. zFS will list the time periods (if infrequent) when the hit ratio was smaller than desired to assist the administrator in determining if the situation is transient or persistent. Another situation where the zFS file cache hit ratio is consistently lower than 90% is when the client applications are aggressively caching outside of zFS. In this case, the zFS is a second-level cache and will generally not see frequent reads of cached data. In these situations the zFS cache hit ratio might be much lower than 90%. Typically, situations where customer applications cache zFS file data aggressively will see file cache hit ratios closer to 50-60%. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Reference documentation: See Performance tuning in z/OS File System Administration. Automation: N/A Routing code: N/A Descriptor code: N/A IOEZH0080I Meta cache total significant intervals = meta_cache_significant % of collected intervals showing sub-optimal performance data = percentage Explanation: Meta cache total significant intervals is the number of observed intervals where meta cache request threshold are met. % of collected intervals showing sub-optimal performance data is the ratio of the total collected intervals with sub-optimal meta cache performance to the meta cache total significant intervals. This message indicates meta cache performance is not optimal. A report of the associated intervals with sub-optimal performance will be displayed following this message. No action is required. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Automation: N/A Reference documentation: See Performance tuning in z/OS File System Administration. Routing code: N/A Descriptor code: N/A IOEZH0083I Found no zFS file system with exception status. Explanation: There is no mounted zFS file system that had: low on space error, disk I/O error, XCF communication error, out of space error, dynamic grow failure, dynamic grow disabled, software error, mounted with no high availability option and RWSHARE, encrypted file system with a non-encrypted log file, containing at least one V4 directory or aggregate disk size exceeded AGGRFULL threshold. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Reference documentation: See zfsadm fsinfo in z/OS File System Administration. Routing code: N/A Descriptor code: N/A IOEZH0084I Found zFS file system(s) in exception state. Explanation Found zFS file systems that meet one of more of the exceptional states, such as low on space, disk I/O error, software error, XCF communication error, out of space error, dynamic grow failure, dynamic grow disabled, mounted with non-use of high availability option and RWSHARE, having at least one V4 directory or aggregate disk size exceeded AGGRFULL threshold. A report of the affected file systems will be displayed after this message. In the report, the file system name, owning system name and the file system status will be shown. The report includes the file system name, owning system name and the file system states. The states contain abbreviated values. For quick reference, the values are defined in a Legend string at the end of the report. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Reference documentation: See Converting an existing v4 directory to an extended (v5) directory in z/OS File System Administration. Automation: N/A Routing code: N/A Descriptor code: N/A IOEZH0088I Could not get completed file system information error =error_code, reason=reason_code. Explanation: Getting file system information is not completed due to the displayed return code and reason code. Health check displays as much information as it can. System action: The system continues processing. Operator response: N/A System programmer response: N/A Problem determination: N/A Module: IOEZHCK1 Source: z/OS File System Automation: N/A Routing code: N/A Descriptor code: N/A The following updates have been made in the z/OS 3.1 IBM Health Checker for z/OS User's Guide SC23-6843-XX In the "Summary of changes for z/OS 3.1", the following new ZFS health checks have been added: "ZFS_CACHE_PERFORMANCE" on page 710 (APAR OA63911, applies to z/OS V3R1 and V2R5) "ZFS_EXCEPTIONS" on page 712 (APAR OA63911, applies to z/OS V3R1 and V2R5). In Part 3. Check Descriptions, the new checks have been added for z/OS File System checks (IBMZFS): ZFS_CACHE_PERFORMANCE ZFS_EXCEPTIONS Chapter 13. IBM Health Checker for z/OS checks In "z/OS File System checks (IBMZFS)", detailed information has been added for the new checks: ZFS_CACHE_PERFORMANCE Description: Use the ZFS_CACHE_PERFORMANCE check to report cases where the zFS file cache hit ratio or meta cache hit ratio are low. The report is based on collected file cache and meta cache performance statistics over a long period of time. The check does not make any specific cache size recommendations. Reason for check: This check provides analysis of performance information that is collected over a long period of time, up to 60 days. If there were intervals that experienced suboptimal performance, the check will display the frequency of occurrence and the dates and times in ranges. zFS considers file cache performance to be good when the file cache hit ratio is above 90%. The meta cache performance is considered to be good when the meta cache hit ratio is above 98%. z/OS releases the check applies to: z/OS V2R5 and z/OS 3.1 with the PTF for APAR OA63911 applied. User override of IBM values: The following sample shows the defaults for customizable values for this check. Use this sample to make permanent customizations in an HZSPRMxx parmlib member that is used at IBM Health Checker for z/OS startup. If you want a one-time only update to the check defaults, omit the first line (ADDREPLACE POLICY) and use the UPDATE statement on a MODIFY hzsproc command. Using non-POLICY UPDATE statements in HZSPRMxx can lead to unexpected results and is not recommended. ADDREPLACE POLICY[(policyname)] [STATEMENT(name)] UPDATE CHECK(IBMZFS,ZFS_CACHE_PERFORMANCE) SEVERITY(LOW) INTERVAL(24:00) DATE('date_of_the_change') REASON('your reason for making change') Debug support: No. Verbose support: No. Parameters accepted: No. Default: None. Reference: For more information about tuning cache sizes to improve performance, see Performance tuning in z/OS File System Administration. Messages: This check issues the following messages: IOEZH0076I IOEZH0077I IOEZH0078I IOEZH0079I IOEZH0080I See z/OS File System Messages and Codes. SECLABEL recommended for multilevel security users: SYSLOW - see z/OS Planning for Multilevel Security and the Common Criteria for information about using security labels. Output: An example of the report is as follows: The following shows an example of cache performance analysis: CHECK(IBMZFS,ZFS_CACHE_PERFORMANCE) SYSPLEX: XXXXXXXX SYSTEM: YYYYYYYY START TIME: 01/30/2023 10:21:05.508504 CHECK DATE: 20220919 CHECK SEVERITY: LOW IOEZH0079I File cache total significant intervals = 40 35% of collected intervals showed sub-optimal performance data. Time periods with sub-optimal performance ------------------------------------------- 01/28/2023 10:32:11 - 01/28/2023 11:32:11 01/29/2023 05:32:11 - 01/29/2023 15:32:11 01/30/2023 01:32:11 - 01/30/2023 04:32:11 IOEZH0080I Meta cache total significant intervals = 41 53% of collected intervals showed sub-optimal performance data. Time periods with sub-optimal performance ------------------------------------------- 01/29/2023 01:32:11 - 01/29/2023 23:32:11 END TIME: 01/30/2023 10:21:05.563131 STATUS: SUCCESSFUL ZFS_EXCEPTIONS Description: ZFS_EXCEPTIONS provides detailed file system information for mounted zFS file systems that have one or more exception states. Examples of the exception states are as follows, with the abbreviations in parentheses. - Low on space error (L) - Disk I/O error (IE) - XCF communication error (CE) - Out of space error (SE) - Software error (DI) - Dynamic grow failure (GF) - Dynamic grow disabled (GD) - Mounted with no high availability option and RWSHARE (NH) - Encrypted file system with a nonencrypted log file (LN) - File system containing at least one V4 directory (D4) - Aggregate disk size exceeded AGGRFULL threshold (AF) The report shows the file system name, owning system, and file system states. The states contain abbreviated values. For quick reference, the values are defined in the Legend section at the end of the report. Tip: Use the z/OS UNIX check USS_CLIENT_MOUNTS to verify that, in a shared file system configuration, sysplex-aware file systems are not sysplex-aware. That is, they were mounted with the RWSHARE option. For more information about the check, see "USS_CLIENT_MOUNTS" on page 629. If file systems containing V4 directories (D4) are displayed in the report, consider converting to V5 directories by using IOECONV4 for better performance. For more information, see Converting an existing v4 directory to an extended (v5) directory in z/OS File System Administration. Reason for check: This check provides a simpler way to alert users about mounted zFS file systems that have exception states. z/OS releases the check applies to: z/OS V2R5 and z/OS 3.1 with the PTF for APAR OA63911 applied. User override of IBM values: The following sample shows the defaults for customizable values for this check. Use this sample to make permanent customizations in an HZSPRMxx parmlib member that is used at IBM Health Checker for z/OS startup. If you want a one-time only update to the check defaults, omit the first line (ADDREPLACE POLICY) and use the UPDATE statement on a MODIFY hzsproc command. Including non-POLICY UPDATE statements in HZSPRMxx might lead to unexpected results and is not recommended. ADDREPLACE POLICY[(policyname)] [STATEMENT(name)] UPDATE CHECK(IBMZFS,ZFS_EXCEPTIONS) SEVERITY(LOW) INTERVAL(ONETIME) DATE('date_of_the_change') REASON('your reason for making change') Debug support: No. Verbose support: No. Parameters accepted: No. Default: None. Reference: For information about zfsadm fsinfo, see zfsadm fsinfo in z/OS File System Administration. For information about how to convert V4 directories to V5, see Converting an existing v4 directory to an extended (v5) directory in z/OS File System Administration. Messages: This check issues the following messages: IOEZH0083I IOEZH0084I IOEZH0088I See z/OS File System Messages and Codes. SECLABEL recommended for multilevel security users: SYSLOW - see z/OS Planning for Multilevel Security and the Common Criteria for information about using security labels. Output: An example of the report is as follows: IOEZH0084I Found zFS file system(s) in exception state. File system name Owner Status --------------------------------------------------------------- PLEX.IMGXXXXX.V4A DCEXXXX1 RW,RS,NE,NC,NH,D4, AF ZFSAGGR.BIGZFS.XYZ.ZTST DCEXXXX2 RW,RS,NE,NC,NH Legend: RW=Read-write RS=Mounted RWSHARE NE=Not encrypted NC=Not compressed D4=Contains V4 Directory AF=Exceeded AGGRFULL threshold NH=Not high availability Note: One or more file systems have V4 directories in them. Consider converting to V5 directories with IOECONV4 REXX exec or utility for better performance. The supported exception states are L, IE, CE, SE, DI, GF, GD, NH, LN, D4 and AF.
APAR Information
APAR number
OA63911
Reported component name
ZFS ZSERIES FIL
Reported component ID
5696EFS00
Reported release
450
Status
CLOSED UR1
PE
NoPE
HIPER
NoHIPER
Special Attention
YesSpecatt / New Function / Xsystem
Submitted date
2022-10-11
Closed date
2024-02-02
Last modified date
2024-03-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UJ94539 UJ94540
Modules/Macros
IOEAGFMT IOEAGSLV IOEDUMPF IOEE0020 IOEFSCM IOEFSKN IOEFSTHD IOEFSUTL IOEZADM IOEZH003 IOEZHADD IOEZHCK1 IOEZHMSG
| SC236887XX | SC236885XX | SC236843XX |
Fix information
Fixed component name
ZFS ZSERIES FIL
Fixed component ID
5696EFS00
Applicable component levels
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU011","label":"Systems - zSystems software"},"Product":{"code":"SG19O"},"Platform":[{"code":"PF054","label":"z Systems"}],"Version":"450"}]
Document Information
Modified date:
03 March 2024