Troubleshooting
Problem
This document discusses issues with lock contention on the QPJOBLOG printer file and the QEZJOBLOG output queue.
Symptom
When many jobs ends at the same time and attempt to writer their job logs to a spooled file, contention or "bottlenecking" can occur on either the QPJOBLOG printer file or the QEZJOBLOG output queue.
Cause
This can occur because exclusive locks are needed on the QPJOBLOG printer file and on the QEZJOBLOG output queue as job logs are written to a spooled file.
Diagnosing The Problem
When contention over the QPJOBLOG printer file is severe enough, it can result in the following error messages:
MCH5802: Lock operation for object QPJOBLOG not satisfied.
CPF3330: Necessary resource not available.
When contention over the QEZJOBLOG output queue is severe enough, it can result in the following error messages:
CPF2528: Job log not written to output queue because of CPF4218.
CPF4218: Output queue &6 in &7 not available.
Note: Output queue contention is not limited to QEZJOBLOG. Some customers funnel all or the majority of spooled output to one or only a few output queues. This can result in bottlenecking that can be much worse than QEZJOBLOG.
Resolving The Problem
There are several recommendations to help alleviate lock contention over the QPJOBLOG printer file, or the QEZJOBLOG output queue:
|
o | Change the logging level of job descriptions to not cut job logs or cut job logs only where needed. Note: If a job ends abnormally, the job log will be cut regardless of the job’s logging level. Decreasing the number of spooled files on QEZJOBLOG will result in the WRKOUTQ OUTQ(QEZJOBLOG) command holding the exclusive lock for a shorter duration. |
| o | Change the QLOGOUTPUT (Job log output) system value or the LOGOUTPUT (Job log output) parameter in your job descriptions to specify either *JOBLOGSVR or *PND. When *JOBEND is specified, the job log will be produced by the job itself. If the job cannot produce its own job log, the job log will be produced by a job log server. When *JOBLOGSVR is specified, the job log will be produce by the job log server. The job will be allowed to end immediately and will go to OUTQ status, so you can display the job log in the resulting QPJOBLOG spooled file. Note that lock contention can still occur with the job log server jobs. When *PND is specified, the job log will not be produced. The job will be allowed to end immediately and will go to JOBLOG PENDING status, but you can display the job log by working with the job and taking Option 10 (Display job log, if active, on job queue, or pending). Individual job logs can also be written to a QPJOBLOG spooled file by changing the job and setting the Job log output (LOGOUTPUT) parameter to *JOBLOGSVR. |
| o | When calling the ENDSBS command, use the ENDSBSOPT(*NOJOBLOG) parameter to reduce the amount of job logs created. |
| o | If experiencing contention over the QPJOBLOG printer file, spread the job log creation using several copies of the QSYS/QPJOBLOG printer file. If experiencing contention over the QEZJOBLOG output queue, spread the job log spooled files out to several output queues. An example of how to route job logs created by different subsystems to a different job log output queue is as follows: 1. Create a library for each subsystem defined on the system that requires a separate job log output queue. 2. Create a duplicate of the QUSRSYS/QEZJOBLOG output queue and put a copy in each library created. 3. Create a duplicate of the QSYS/QPJOBLOG printer file and put a copy in each library created. 4. Change the OUTQ attribute of each duplicated QPJOBLOG printer file from QUSRSYS/QEZJOBLOG to *LIBL/QEZJOBLOG. 5. Create a routing entry for each subsystem to call separate programs that do a CHGSYSLIBL LIB(x) so that the system portion of the library list is modified to have the new libraries created in step 1 above QSYS. Note: When using this technique, be aware that automatic cleanup of system output will not be done on these duplicated output queues. |
| o | Ensure that the system is tuned correctly. For operation such as WRKOUTQ, paging throughput is a significant gating factor. Increase resources such as memory that is allocated to jobs accessing job logs. |
| o | At off peak times when the output queue is not in use, move spooled files to alternate output queues. An application like DLTOLDSPLF could be modified to move spooled files older than X number of days using the CHGSPLFA command (instead of DLTSPLF). |
| o | Decrease the retention period for job logs using OA cleanup functions (GO CLEANUP). |
| o | To reduce contention on QEZJOBLOG, use the WRKSPLF or WRKJOB OPTION(*SPLF) instead of WRKOUTQ to access job log spooled files. |
| o | Ensure that QEZJOBLOG is only being used to hold job log spooled files. |
| o | Ensure that the QRCLSPLSTG system value is not set to *NONE. Using *NONE can adversely affect the performance of creating a spooled file. |
Spool Performance Experience Report
IBM Development recommends that users look through the Spool Performance Considerations Experience Report when experiencing problems related to spool performance. This experience report will:
| o | Identify spooled objects that are most likely to encounter lock contention. |
| o | Provide information about messages that can be expected if performance bottlenecks are encountered. |
| o | Identify several scenarios that can cause spool performance problems. |
| o | Provide recommendations to reduce specific spool performance issues. |
The Spool Performance Considerations Experience Report, which is available in the IBM Knowledge Center at:
Information Concerning Pending Job Logs
Pending job logs will remain available until either the pending job log is removed or the job log is written to a QPJOBLOG spooled file. If GO CLEANUP is used to remove job logs and other system output after a specified number of days, then it will also remove pending job logs for jobs that completed more than the specified number of days ago.
Pending job logs can be displayed by using the Work with Job (WRKJOB) command and taking Option 10 (Display job log, if active, on job queue, or pending), or by using the Work with Job Logs (WRKJOBLOG) command and taking Option 5 (Display).
Pending job logs can be written to a QPJOBLOG spooled file through System i Navigator, or by using the Work with Job Logs (WRKJOBLOG) command, taking Option 12 (Work with job), taking Option 40 (Change job), and then setting the Job log output (LOGOUTPUT) parameter to *JOBEND. You can also use the Change Job (CHGJOB) command, for example:
CHGJOB JOB(job-number/job-user/job-name) LOGOUTPUT(*JOBEND)
For more information, please refer to the following Technote documents:
N1014783: Jobs in Joblog Pending Status; Joblogs Are Not Being Created
N1018808: Removing Pending Job Logs
as well as the Job log pending topic in the IBM i Information Center:
Information Concerning the QLOGOUTPUT System Value
Setting the QLOGOUTPUT (Job log output) system value to *PND, or setting the Job log output (LOGOUTPUT) parameter in a job description or in the Change Job (CHGJOB) command to *PND will cause the job log to remain associated with the job without being written to a QPJOBLOG spooled file. In this case, job logs will continue to take up DASD, just the DASD used will not be within the QSPL data base.
The QLOGOUTPUT system value and LOGOUTPUT parameter can be set to *JOBLOGSVR instead of *PND. In this case, job logs will be written to QPJOBLOG spooled files by the job log server, not by each job as it is ended. The use of *JOBLOGSVR will not help reduce the amount of DASD used to store spooled files, but can help where there are lock contention issues on either the QPJOBLOG printer file or the QEZJOBLOG output queue.
For more information, please refer to the Jobs system values: Produce printer output for job log topic in the IBM Knowledge Center:
Was this topic helpful?
Document Information
Modified date:
18 December 2019
UID
nas8N1020404