Introduction
In order to discover software available to a z/OS operating system image, the Inquisitor component (program HSIPINQ – often referred to as the IQ) of Tivoli Asset Discovery for z/OS must extract data from the DASD subsystem accessible to the system being examined. While extensive filter controls are available for use, the usual case is that a complete scan of the DASD subsystem is performed. This usually collects a substantial volume of data, which is in turn processed by IQ Import which stores relevant information from the IQ scan into the data base.
While the SCANDIR process of program HSIPINQ is fairly light on resource consumption, scanning the entire DASD subsystem can take prolonged periods of elapsed time. Scan durations exceeding a day have been experienced on some customer systems. The main cause of the extended elapsed time is the structure and logic flow of the HSIPINQ program. The entire scan process proceeds as a sequential process under the control of a single dispatchable unit (or task).
One of the enhancements delivered in Tivoli Asset Discovery for z/OS Version 8 Release 1 is an Inquisitor program capable of using multiple concurrent units of work (multiple tasks) to perform parallel processing, thereby achieving reduced scan elapsed times.
Multitasking is only employed when scanning DASD by volume, and not when scanning using the catalog. A volume-scanning subtask scans one volume at a time. Each volume is only scanned by one subtask. When a subtask completes a volume scan, it scans the next unscanned volume selected by the request. Subtasks terminate when they find that all volumes selected by the request have been or are being scanned.
The bulk of the reduction in elapsed time occurs because many VTOCs are scanned concurrently. This is especially important when the VTOC sizes used on a system are large. It takes a relatively long time to scan thousands of volumes with 200 cylinder VTOCs when processing only one volume at a time, as customer performance observational data can attest.
Although the use of multitasking does reduce the elapsed time of scans, several other consequences also flow from it.
Region size
Firstly consider data flow. The output of the Inquisitor is a sequential file where the order of records is significant. For example, records about individual programs follow the record about the program library in which they reside. This means that the output must be ordered. Records cannot be written by each subtask as soon as there is a record ready for writing. The data from subtasks must be coordinated by the main task which is also responsible for writing the relevant data to the output file, which is usually the HSIPZIP file.
To coordinate the data from all of the subtasks, the data from the subtasks is staged in the virtual storage of the job's region. So, a larger region is required by the V8R1 Inquisitor when compared to the V7R5 Inquisitor.
The volume-scanning subtasks discover program libraries which are subsequently scanned by the main task. To further reduce the total elapsed time of SCANDIR requests, the subtasks can also scan PDS (but not PDSE) libraries. (PDSE libraries are always scanned by the main task.) This adds significantly to use of region storage, given that it is easily possible for a single PDS to contain thousands of members, and that the data from thousands of PDS libraries may need to be staged in the region as the main task works through the data accumulated by the subtasks.
S878 and SC78 abends are indicative of insufficient region size, and represent occasions where data was lost (resulting in an incomplete scan) even if the Inquisitor job step does not abend.
The LIBMAINTASK keyword can be used to ensure that all scanning of libraries discovered by the subtasks is performed by the main task.
The multitasking level of the Inquisitor should be kept low enough so that the region size has the capacity to hold the data collected by the subtasks as it is staged to be processed by the main task.
I/O load
No matter how it is scheduled, any particular DASD scan requires a certain number of I/O operations. The multitasking capability of the Inquisitor was introduced to reduce the elapsed time over which these I/O operations occurred. A greater number of concurrent I/O operations is required to achieve a smaller elapsed time. As this concurrency increases, there often comes a point where the load on the DASD subsystem impacts on the performance of other workloads.
Symptoms flowing from an excessive I/O load may include spikes in the utilization of device, control unit, and channel, as well as increased queuing at each of those levels as well is in z/OS itself. Erratic application responsiveness and interference to the normal flow of demand paging and application I/O flow can be additional symptoms of I/O stress.
The multitasking level of the Inquisitor should be kept low enough so that other workloads are not unduly impacted by the extra I/O load on the DASD subsystem.
CPU absorption
The effectiveness of multitasking relies on CPU-I/O overlap. Inquisitor subtasks can schedule I/O operations to fetch data from a DASD volume, and while waiting for the data to be delivered by the I/O subsystem other subtasks can be dispatched, which can in turn then schedule their I/O operations and wait for the results.
If the available CPU resources are insufficient in number and/or speed, the benefits of additional subtasks are limited, and may even be negative. This happens when the number of dispatchable units (subtasks) is too large to be serviced by the logical processors within the duration of I/O operations.
To look at it another way, an LPAR with 10 uncapped processors might easily be able to keep 90 out of 100 subtasks waiting for I/O completion most of the time, but an LPAR with one or two severely capped processors might struggle to keep one or two subtasks waiting for I/O completion, especially if there a many other subtasks which have not yet fully initialized. (Subtask initialization may include acquisition of working storage, dynamic device, opening file DCBs, acquisition of I/O buffers, etc. This is in addition to all the work the operating system must do to establish a large number of subtasks in an address space which is an intrinsically serialized process.)
If CPU resources are such that only a small number of subtasks can be kept productive concurrently, then adding more subtasks not only increases CPU time, but also increases total elapsed time.
The multitasking level of the Inquisitor should be kept low enough so that available CPU resources can effectively dispatch each and every subtask. Subtasks which do not contribute to throughput in a positive way represent extra overhead.
Controlling the multitasking level
The MAXTASKS operand has been introduced to control the level of multitasking. This operand is a property of individual requests. The MAXTASKS value can be set independently for each request.
For example, if a particular z/OS system's DASD was made up from two SAN (Storage Area Network) boxes, one with a small concurrent I/O capability and one with a large concurrent I/O capability, then the DASD scan could be implemented with 2 SCANDIR requests, each with a MAXTASKS value appropriate for the relevant SAN box. (In such a scenario, the DEVICE and/or XDEVICE operands would probably be used to control the volume selection of each request.)
Refer to
for SCANDIR operand function and syntax.
Currently, the HSISINQZ sample job shipped in the SHSISAMP library specifies the recommended value of 10, and the program default (the value used when the MAXTASKS operand is not specified) is also 10. Because each VTOC is scanned by a single task, the value of MAXTASKS used will not exceed the number of volumes selected for scanning.
[Note that the V8R1 GA program default value for MAXTASKS was 200. This was changed to 10 by PTF UA74298 which was issued in July 2014 to close APAR OA45545.]
Effect of MAXTASKS on VTOC scanning
The Inquisitor was run using different MAXTASKS values to illustrate the effect of the MAXTASKS value on performance. The systems was running z/OS V2R1 on a zEC12 in an LPAR with 3 CPUs. The DASD scan included 1318 3390 volumes with the VTOC sizes totalling 142447 tracks.
The first series of tests used a data set name selection mask of NOTEXIST.* to prevent any program libraries from being scanned. As a result, the effect MAXTASKS has on VTOC scanning performance and the underlying REGION requirements can be seen.
The data from a run of Inquisitor from V7R5 is plotted as a zero MAXTASKS value.
Deviation from smooth curves can be attributed to interference from other workloads, including activity on other systems in the SYSPLEX using the same DASD subsystem.
Changing MAXTASKS has little effect on CPU time, although IQ V7R5 used 15% more CPU than IQ V8R1. As MAXTASKS approached its maximum value, TCB time did climb slightly. Almost all of the CPU time was consumed by the subtasks because – since all data sets were effectively excluded – no work was queued to the main task for processing, leaving only initialization and termination processing for it to perform.
The VIRT, SYS, EXT and ESYS figures are taken directly from the IEF032I messages. The linear correlation between the size of high private storage and the number of subtasks can be seen. The BCP does a good job of keep most LSQA and SWA control blocks above the line, as evidenced by the steeper gradient of the ESYS line when compared to the SYS line. The region storage usage rises slowly as more copies of working storage (which is below the line because it contains the DCB for the VTOC) are instanced. With no program libraries selected the extended region requirements remain constant.
The second series of tests performed a full SCANDIR data collection run. Again, the V7R5 Inquisitor is plotted as having a MAXTASKS value of zero. All jobs used about the same CPU time. The single most striking difference is the growth of the extended region.
On this system, the program library distribution across the DASD was such that once MAXTASKS was around 50, adding more subtasks was not able to queue up work for the main task any faster. The fact that elapsed time is still trending downwards may mean that DASD with higher device numbers (that is, those volumes processed near the end of the scan) contained few program libraries and so could be scanned more rapidly by the extra subtasks, but there were no further programs to be discovered to add to the backlog of the main task.
For much larger software inventories a more linear consumption of the extended region with MAXTASKS value could reasonably be expected. It may even be possible to exhaust the available 2 gigabyte address space, in which case MAXTASKS would have to be reduced.
Reasons for using a larger value of MAXTASKS
Environmental conditions suggesting that a larger value of MAXTASKS may be advantageous are:
- A large number of DASD volumes to scan.
- Large VTOC sizes.
- DASD hardware capable of handling large numbers of parallel I/O requests.
- z/OS storage configuration more than adequate for large region sizes.
- z/OS CPU resources include numerous and/or fast CPUs with spare capacity.
- z/OS workloads allow Inquisitor batch job tasks to be dispatched quickly when ready.
Reasons for using a smaller value of MAXTASKS
Environmental conditions suggesting that a smaller value of MAXTASKS may be preferable are:
- VTOC sizes are generally no more than a few cylinders.
- DASD hardware cannot deliver high volume parallel I/O.
- Performance of existing applications degrades when Inquisitor is executing.
- Large region sizes cause paging which impacts more important workloads.
- z/OS CPU resources are not sufficient to productively dispatch a large number of Inquisitor subtasks.