Rename failures

If there is a failure during the optional rename step of coordinated change master key or coordinated refresh, CSFM629I and CSFM630I messages will be written to the ICSF job log to indicate the reason for the failure.

The rename function uses the IDCAMS processor to perform the actual VSAM data set rename. CSFM629I messages are used to route IDCAMS processor messages to the ICSF job log when the IDCAMS processor fails to perform the rename. The CSFM629I messages contain the reason from the IDCAMS failure. These messages are followed by a CSFM630I message that indicates which data set name failed to be renamed to which new name.

ICSF KDS data sets are KDS VSAM data sets. They consist of 3 parts: a cluster name, an index name, and a data name. For example, if you use the sample JCL provided in the "Steps to create the CKDS" section of the z/OS Cryptographic Services ICSF System Programmer's Guide, the cluster name, data name, and index name will be the following in order.

CSF.CSFCKDS
CSF.CSFCKDS.DATA
CSF.CSFCKDS.INDEX

When the rename option is selected, all 3 parts of the active KDS will be renamed to the archive name, and all 3 parts of the target KDS will be renamed to the active name. When renaming the data and index portions of a KDS VSAM data set, the suffix format of the original data set is maintained. For example, if the preceding data set names are used for the active CKDS, and the archive data set name is specified as CSF.CSFCKDS.ARC, the 3 portions of the active CKDS will be renamed to:

CSF.CSFCKDS.ARC
CSF.CSFCKDS.ARC.DATA
CSF.CSFCKDS.ARC.INDEX

In the case of a failure during rename processing, the coordinated function will attempt to back out and rename the data sets back to their original names. If the back out fails, you may end up with a partially renamed data set. This can be easily corrected by performing an IDCAMS ALTER from JCL.

Whenever a rename failure occurs, scan the ICSF job log of the initiating system for CSFM629I and CSFM630I messages. These messages will indicate which data set part failed during rename and if backout processing was able to rename the data set back to its original name. If backout processing was able to rename back to the original name, check the catalog for the data set name that failed to be used for rename. Most likely you have a conflict with the archive data set name and need to either rename existing data sets in your catalog or choose a different archive name.

If backout processing failed to rename your data set back to the original name, use ISPF to confirm that the data set parts match up with what is reported in the CSFM629I and CSFM630I messages.

For example, if during a coordinated CKDS change master key operation, the active CKDS cluster name is successfully renamed to the archive name, but the data portion fails to be renamed, backout processing begins. If backout processing fails to rename the data set back to the original active CKDS name, ICSF will shut down all instances in the sysplex CKDS cluster because the active CKDS name is only half renamed. In this scenario, the following set of messages may be reported in the ICSF job log.

CSFM618I CKDS DATA SET CSF.CSFCKDS RENAMED TO CSF.CSFCKDS.ARC
CSFM629I IDCAMS  SYSTEM SERVICES                                     TIME: 13:35:12        06/07/11     
CSFM629I
CSFM629I  ALTER CSF.CSFCKDS.DATA                    -
CSFM629I  NEWNAME(CSF.CSFCKDS.ARC.DATA               )
CSFM629I IDC3013I DUPLICATE DATA SET NAME
CSFM629I IDC3009I ** VSAM CATALOG RETURN CODE IS 8 - REASON CODE IS IGG0CLE6-8
CSFM629I IDC0532I **ENTRY CSF.CSFCKDS.DATA NOT ALTERED
CSFM629I IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 8
CSFM629I
CSFM629I IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 8
CSFM630I CKDS RENAME FAILED: CSF.CSFCKDS.DATA TO CSF.CSFCKDS.ARC.DATA
CSFM629I IDCAMS  SYSTEM SERVICES                                     TIME: 13:35:18        06/07/11     
CSFM629I
CSFM629I  ALTER CSF.CSFCKDS.ARC                     -
CSFM629I  NEWNAME(CSF.CSFCKDS                        )
CSFM629I IDC3013I DUPLICATE DATA SET NAME
CSFM629I IDC3009I ** VSAM CATALOG RETURN CODE IS 8 - REASON CODE IS IGG0CLE6-8
CSFM629I IDC0532I **ENTRY CSF.CSFCKDS.ARC NOT ALTERED
CSFM629I IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 8
CSFM629I
CSFM629I IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 8
CSFM630I CKDS RENAME FAILED: CSF.CSFCKDS.ARC TO CSF.CSFCKDS
CSFM620I COORDINATED CHANGE-MK MAINLINE PROCESSING FAILED BECAUSE THE ACTIVE DATA SET 
         CANNOT BE RENAMED TO THE ARCHIVE NAME.
CSFM622I COORDINATED CHANGE-MK PROGRESS: BACKOUT IS BEING DRIVEN BY MAINLINE.
CSFM629I IDCAMS  SYSTEM SERVICES                                     TIME: 13:35:24        06/07/11     
CSFM629I
CSFM629I  ALTER CSF.CSFCKDS.ARC                     -
CSFM629I  NEWNAME(CSF.CSFCKDS                        )
CSFM629I IDC3013I DUPLICATE DATA SET NAME
CSFM629I IDC3009I ** VSAM CATALOG RETURN CODE IS 8 - REASON CODE IS IGG0CLE6-8
CSFM629I IDC0532I **ENTRY CSF.CSFCKDS.ARC NOT ALTERED
CSFM629I IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 8
CSFM629I
CSFM629I IDC0002I IDCAMS PROCESSING COMPLETE. MAXIMUM CONDITION CODE WAS 8
CSFM630I CKDS RENAME FAILED: CSF.CSFCKDS.ARC TO CSF.CSFCKDS
CSFM621I COORDINATED CHANGE-MK BACK OUT PROCESSING FAILED BECAUSE THE ARCHIVE DATA SET
         CANNOT BE RENAMED TO THE ACTIVE NAME.
CSFM621I COORDINATED CHANGE-MK BACK OUT PROCESSING FAILED BECAUSE ICSF IS UNABLE TO 
         RELIABLY RESTORE THE ORIGINAL ACTIVE KDS.
CSFM622I COORDINATED CHANGE-MK PROGRESS: CANCELING CORE WORK.
CSFM616I COORDINATED CHANGE-MK FAILED, RC=0000000C RS= 00000C3E SUPRC= 0000000C SUPRS= 
         00000C42 FLAGS= C5800000.
CSFU006I CHANGE-MK FEEDBACK: CC=0000000C RSN=00000C3E SUPPRC=0000000C SUPPRSN=00000C42 
         FLAGS=C5800000.
CSFM308I MEMBER XXX REPORTED REMOVED FROM SYSPLEX GROUP SYSICSF.
CSFM308I MEMBER XXX REPORTED REMOVED FROM SYSPLEX GROUP SYSICSF.
CSFM401I CRYPTOGRAPHY - SERVICES ARE NO LONGER AVAILABLE.

This sequence of messages indicates that the active CKDS name of CSF.CSFCKDS was renamed to CSF.CSFCKDS.ARC. Then, the CSF.CSFCKDS.DATA data portion of the active CKDS failed to be renamed to CSF.CSFCKDS.DATA.ARC because another data set in the catalog was already using this name. At this point, the coordinated CKDS change master key function tried to back out and rename the cluster portion of the CKDS from CSF.CSFCKDS.ARC back to its original name of CSF.CSFCKDS. However the renaming failed because another data set with name CSF.CSFCKDS now exists in the catalog. The result is a half renamed active CKDS which causes ICSF to shut down across the CKDS sysplex cluster.

The first step to resolving this problem is to confirm in ISPF that the following data set names reported in the messages above do exist:

CSF.CSFCKDS.ARC
CSF.CSFCKDS.DATA
CSF.CSFCKDS.INDEX

Once this is confirmed, the next step is to rename the cluster name back to the original name manually by calling IDCAMS ALTER from JCL. Before doing that, the messages above indicate that back out processing already failed to rename the cluster name back because another data set is now using that name. The data set that has taken that name should be renamed to a different name as this name is needed to restore the active CKDS.

Once the cluster name conflict has been resolved, issue IDCAMS ALTER from JCL to rename the CSF.CSFCKDS.ARC cluster name back to the original active CKDS name of CSF.CSFCKDS.

For example:

//DEFINE  EXEC PGM=IDCAMS,REGION=4M
//SYSPRINT DD  SYSOUT=*
//SYSIN DD *
    ALTER CSF.CSFCKDS.ARC  -
          NEWNAME(CSF.CSFCKDS)
/*

ICSF may be restarted on all instances that were previously taken down. Processing should resume as normal and the coordinated CKDS change master key with rename option may be issued again with an archive data set name that does not have a conflict in the catalog.