QMSF Jobs Fail with a CPI9A9C Errors with QAOK* Files

Resolving The Problem

Sometimes the QMSF jobs end unexpectedly with a message CPI9A9C error, Search data does not exist. Generally, this problem has been observed following an upgrade from CISC to RISC where the user had to do a restore of objects into the QUSRSYS library. Or any upgrade without restores of QUSRSYS involved immediately following the upgrade, but when the System Distribution Directory (SDD) Files were already in a somewhat invalid state preceding the upgrade (for example, something left over from a previous release). At other times, any restore into the QUSRSYS library or any restore of user profiles may trigger the problem as well.

The recovery text in the CPI9A9C informational message says to invoke the CHGSYSDIRA CL command specifying *YES in the ALWSCH (Allow Search) keyword. This suggestion is correct and should be attempted before anything else. However, there are some cases when the user prompts the CHGSYSDIRA CL command and the ALWSCH keyword already shows up with a *YES value. In that case, to force the recreation of the Directory Search data, it is necessary to issue CHGSYSDIRA ALWSCH(*NO) first, followed by a CHGSYSDIRA ALWSCH(*YES). Note that this command, especially when the ALWSCH(*YES) clause is specified, requires that some of the SDD files be used by no other job. Therefore, is strongly recommended that this is done in restricted state to avoid any conflicts.

Before restarting QMSF, a quick way to determine if the CHGSYSDIRA invocations specified earlier worked is to run one of the following commands. Using the WRKDIRE or DSPDIRE command, press F10 (Search directory). Search for a user ID (for example, QDFTOWN) that should be in the directory. The search should work okay and no message CPI9A9C, Search data does not exist, should be issued. However, if the message CPI9A9C message is shown at the bottom of the Search directory panel again, a problem persists with the SDD data. In that case, it is very likely that the Directory files have somehow been corrupted. Go to the Fixing up QAOKL* and QAOKP* System Distribution Directory Files section for detailed steps on how to determine that this is the case. If this is not the case, there is another problem. Report it to the System Distribution Directory developers.

Title: Fixing up QAOKL* and QAOKP* System Distribution Directory Files in QUSRSYS

The System Distribution Directory (SDD) has been around for a long while. Its supported commands, such as ADDDIRE, CHGDIRE, WRKDIRE, CHGSYSDIRA, to name a few, have not undergone major changes since the V3R1M0 timeframe. Therefore, if all of a sudden these commands start exhibiting strange behavior and function checking in the simplest of operations, the first thing to check is the integrity of its logical and physical files. In some rare instances, the same applies to the Directory Search API (QOKSCHD) which is widely used by the Mail Framework, SMTP and POP3 operating system functions (this list is by no means complete). If this API issues the informational message CPI9A9C indicating that the Search Data does not exist even after the SDD's search data has been rebuilt using the CHGSYSDIRA CL command, verify the integrity of the SDD's files. The SDD files are in QUSRSYS. The logical files are pre-fixed by QAOKL*, and the physical files are pre-fixed by QAOKP*. In addition, there is another file also in QUSRSYS named QAOZL03A. That is an Office Vision/400 logical file that is built over some of the QAOKP* files that. For the purpose of this writing, must be considered as part of the SDD files even though it is used and owned by Office Vision/400 for Personal Directories.

To determine if there is a problem with the QAOK* files, do the following:

o

WRKF FILE(QUSRSYS/QAOK*) If this panel shows several times that look similar to QAOKLxxxxxx(3) or QAOKPxxxxxx where xxxxxx is some sort of consecutive number and the text description reads similar to "Old name for QAOKL...." or "Old name for QAOKP...." respectively, the integrity of the files may be in question. Therefore, the recovery steps should be taken. If this is not the case, the problem may be a legitimate problem. Report it to the System Distribution Directory developers.

o

Following is an example of what you can expect to see when things are not quite right:

Work with Files                                                                                                               Type options, press Enter.                                                      1=Create   3=Copy   4=Delete   5=Display physical file member                  8=Display file description     9=Save   10=Restore   13=Change description                                                                                 Opt  File        Library     Attribute   Text                                                                                                                     QAOKLABA    QUSRSYS     LF          OS/400 System Directory File               QAOKLAKA    QUSRSYS     LF          OS/400 System Directory File      QAOKLA0001  QUSRSYS     LF          Old name QAOKLABA in QUSRSYS owned by     QAOKLA0002  QUSRSYS     LF          Old name QAOKLABA in QUSRSYS owned by     QAOKLCLA    QUSRSYS     LF          OS/400 System Directory File              QAOKLC0001  QUSRSYS     LF          Old name QAOKLCLA in QUSRSYS owned by     QAOKLDKA    QUSRSYS     LF          OS/400 System Directory File      QAOKLD0001  QUSRSYS     LF          Old name QAOKLDKA in QUSRSYS owned by     QAOKLEAA    QUSRSYS     LF          OS/400 System Directory File                QAOKLE0001  QUSRSYS     LF          Old name QAOKLEAA in QUSRSYS owned by                                                                                 More... Parameters for options 1, 3, 4, 5, 8, 9, 10 and 13 or command                   ===>                                                                           F3=Exit      F4=Prompt   F5=Refresh   F9=Retrieve   F11=Display names only     F12=Cancel   F16=Repeat position to   F17=Position to

The names with the "Old name ..." text at the left are not right. Names such as "QAOKLABA" are okay. Notice that the correct names always end with an "A" and are 8 characters long. The same thing applies to the physical files as shown below:

Work with Files                                                                                                               Type options, press Enter.                                                      1=Create   3=Copy   4=Delete   5=Display physical file member                  8=Display file description     9=Save   10=Restore   13=Change description                                                                                 Opt  File        Library     Attribute   Text                                                                                                                     QAOKPCLA    QUSRSYS     PF          OS/400 System Directory File      QAOKPC0001  QUSRSYS     PF          Old name QAOKPCLA in QUSRSYS owned by     QAOKPLCA    QUSRSYS     PF          OS/400 System Directory File     QAOKPLGA    QUSRSYS     PF          OS/400 System Directory File      QAOKPL0001  QUSRSYS     PF          Old name QAOKPLCA in QUSRSYS owned by      QAOKPL0002  QUSRSYS     PF          Old name QAOKPLGA in QUSRSYS owned by     QAOKPSPA    QUSRSYS     PF          OS/400 System Directory File     QAOKPSRA    QUSRSYS     PF          OS/400 System Directory File      QAOKPS0001  QUSRSYS     PF          Old name QAOKPSPA in QUSRSYS owned by      QAOKPS0002  QUSRSYS     PF          Old name QAOKPSRA in QUSRSYS owned by                                                                                 More... Parameters for options 1, 3, 4, 5, 8, 9, 10 and 13 or command                   ===>                                                                            F3=Exit      F4=Prompt   F5=Refresh   F9=Retrieve   F11=Display names only     F12=Cancel   F16=Repeat position to   F17=Position to

Once a potential problem has been detected with the QAOK* files, do the following:

1.	The cleanest way of fixing this problem requires that the user have a backup tape of his QUSRSYS/QAOK* files available. A backup tape of all the QUSRSYS library will do, too. This backup tape needs to be complete (for example, at least all the QAOK* files and QAOZL03A files must be on that tape. In addition, this tape must not include any of the QAOKLxxxxxx or QAOKPxxxxxx files, where the xxxxxx is the weird numeric suffix). This backup tape must be CURRENT. Otherwise some loss of data will be inevitable. Because the files are logically connected to each other, restoring partially (for example, only some files) will not do. At the very least, the 15 physical files (containing the actual user data) must be on the backup tape. The logical files can always be recreated when re-installing QUSRSYS (these files are only IBM-supplied templates or views, they contain no user data).
2.	Get the system in restricted state.
3.	Remove all the QUSRSYS/QAOKL* files and remove the QUSRSYS/QA0ZL03A file too.
4.	Remove all the QUSRSYS/QAOKP* files.
5.	Restore from the backup tape all QAOKP* and QAOKL* files into QUSRSYS: RSTOBJ OBJ(QAOKP* QAOKL*) SAVLIB(QUSRSYS) ...
6.	Restore from the backup tape the QAOZL03A file.
7.	As of this writing (V4R5M0), there are only 15 physical QAOKP* files and 32 logical QAOKL* files. This is very unlikely to change on releases beyond that.
8.	Invoke the INZSYS CL command.
9.	Rebuild the search data file. First, remove all search data records by issuing CHGSYSDIRA ALWSCH(*NO). Make sure this command completes successfully. Issue the command one more time but now recreate the search data records: CHGSYSDIRA ALWSCH(*YES). Make sure this invocation completes successfully.
10.	To verify Step 7, run the WRKDIRE or DSPDIRE command, and press F10 (Search Directory). Fill in a USRID value that exists in the System Distribution Directory, and try to search for it. The match should be found.
11.	To verify everything is back to normal, re-issue the SDD command (for example, ADDDIRE or CHGDIRE) that failed earlier. It should work now. Or, if the problem was initially detected in the QMSF jobs, re-start the QMSF jobs now.

Notes

o	The SDD files have some indirect connections to IBM OfficeVision/400 (OV/400) and DLOs. Restoring from a backup may cause slight problems with OV/400 or DLOs. For example, if directory entries were added after the backup, after the restore, OV/400 or documents could end up referring to directory entries that do not exist (at least not until the user adds them back to the SDD).
o	Since files are going to be deleted from QUSRSYS, for safety it would be a good idea to backup QUSRSYS from the system on which the deletes are going to be done. This is just in case something else gets removed by mistake.
o	At any time if there is a need to check the contents of the physical files (for example, QAOKP* files), use the DSPPFM CL command to accomplish this. If you need to display logical files' (for example, QAOKL* files) relationships to physical files, use the DSPDBR CL command.
o	And finally, the above is recommended only as the last resort. And only when the SDD files WRKF panel looks like the panels shown earlier and, most importantly, when some SDD functions (or in some rare instances the QMSF jobs only when the CPI9A9C error is involved) fail to work correctly or do not work at all.

Why Do These Weird-Named Files Get Created in the First Place?

These (for simplicity, let us label them OBSOLETE) files are the result of a restore operation of FILE objects on the media that already exist on the target system. When object differences are allowed in the RESTORE (which most of the times will be the case), the FILE objects on the media are in the majority of cases the ones that prevail on the target system and the existing FILE objects on the target system get renamed to the "Old name for ..." type-names. Note that this applies to all file objects, not only the System Distribution Directory files.

Now when doing this type of restore on the System Distribution Directory files, it is best to check immediately after the restore completes that no such OBSOLETE files have been generated. If these obsolete files are present, one should immediately remove them, the logical obsolete ones first followed by the obsolete physical ones. This must be done before trying to use any of the files involved. If all goes well, then that is the end of it. However, if some of the obsolete physical files cannot be deleted because they have connections to some non-obsolete logical file(s), then the restore has to be retried. But first, ALL QAOKL* and QAOKP* files must be removed from the target system. Only then, the restore of all QAOKL*⁽¹⁾ and QAOKP* from the media can proceed.

This is the best way to safely RESTORE the QAOKL* (see Note 1) and QAOKP* files into QUSRSYS any time, whether it is when moving the SDD data from an old system to a new system, or doing just an every day restore to recover from some disaster scenario or loss. The OS/400 Backup and Recovery Guide (V4R3 version: publication # QB3ALE02) has tons of information on the subject. Refer in particular to the section under Chapter 17 How to Restore Specific Types of Information, subtitle Restoring Database Files and Comparing File Attributes During a Restore Operation. This is very general, geared to all database files; however, the System Distribution Directory files are just regular database files. Those rules apply to them too.

Finally note that in some cases, even with the presence of these Obsolete files the System Distribution Directory functions will work fine. This is possible if the logical files did not get hopelessly corrupted when all this renaming took place. There is no way to anticipate this. In fact, the problem may be started on one release, go unnoticed for the life of that release and become noticeable later immediately after the user moves on to the next release.

Preventing This from Happening

The generation of the obsolete files can only happen at restore operations. So, the user should be advised whenever restoring the SDD files over onto existing files on the target system, to first remove ALL QAOKL*⁽¹⁾ and ALL QAOKP* files from the target system and only then proceed with the restore. This is especially true when the restore is done specifying the parameter that ALLOWS object differences. Otherwise, obsolete files can result from this type of restore.

1.	The QAOZL03A is also included in this category.
2.	If the backup has only the 15 physical files (and/or the 32 logical files are not complete), restore only the physical files. Before moving on to Step 7, reinstall QUSRSYS (for example, GO LICPGM and install QUSRSYS). This will recreate all missing logical files. Then move on to Step 7 and continue on as indicated above.
3.	The QAOZL03A file could also get renamed to QAOZLxxxxxx.
4.	The QAOZL03A may not always exist on the customer system. If the file does not exist at all on its system, there is no need to verify that the file is in the backup tape and there is no need to restore this file.