IT46957: IBM STORAGE COPY DATA MANAGEMENT MAINTENANCE JOB LEAVES BEHIND IBM FLASHCOPY CONSISTENCY GROUPS

APAR status

• Closed as program error.

Error description

• When flash copies within a consistency group reach their
  retention period.
  The IBM Storage Copy Data Management maintenance job will delete
  the flash copies within the consistency group and then proceed
  with deleting the empty consistency group.
  
  When the same SLA policy is used for multiple backup jobs.  An
  issue can occur where the maintenance job removes the flash
  copies from the consistency group but then leaves behind the
  empty consistency group on the IBM SVC storage.
  There are no log messages within the CDM appliance that
  indicates this issue is happening.
  
  Due to this issue, the maximum consistency group limit on IBM
  SVC storage can be reached.  Resulting in failed CDM application
  backups with the following error message.
  ERROR,[<timestamp>],2,Unable to get consistency group. Error:
  The action failed because the maximum number of objects has been
  reached.
  ERROR,[<timestamp>],2,Command mkfcconsistgrp -name
  "##_##_SQL_FCNC_##_##_##_##" failed with exception com.catalogi
  c.ecx.ibmsvcprovider.exception.IBMSVCExecCommandException:CMMVC6
  objects has been reached.
  
  
  Affected versions: IBM Spectrum Copy Data Management 2.2.22.x
  

Local fix

• 1) Manually remove the left behind empty consistency groups on
  the IBM SVC storage so that backups can proceed.
  
  2) For backups that are affected by this issue do the following.
  
    1. Select to hold the schedule of the affected backup jobs.
    2. Create a new SLA and assign it to a new job definition that
  will replace the affected original job.  The new job must have a
  one-to-one relationship with the SLA.  Do not add the SLA to
  another job definition.
    3. After all of the restore points from the original job have
  been expired, delete the job from CDM.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * No                                                           *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * When flashcopies within a consistency group reach their      *
  * retention period.                                            *
  * The IBM Storage Copy Data Management maintenance job will    *
  * delete the flashcopies within the consistency group and then *
  * proceed with deleting the empty consistency group.           *
  *                                                              *
  * An issue can occur where the maintenance job removes the     *
  * flashopies from the consistency group but then leaves behind *
  * the empty consistency group on the IBM SVC storage.          *
  * There are no log messages within the CDM appliance that      *
  * indicates this issue is happening                            *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• This maintenance fix checks for orphaned empty consistency
  groups based on these three parameters (prefix label, jobid,
  SLAid) It checks this against active running jobs, that have
  flash copies, which need to be deleted. This check is against
  the catalog of the SCDM server itself.
  When the consistency group is created it follows the naming
  convention below which is then cataloged into the SCDM Server.
  Prefix Label_Jobid_SLAid_consistency.grp.id --> ex:
  Keep2Snapshot_KP_TEST_1005_1001_<consistency_group_id>
  When the maintenance job runs. It will check for any empty
  consistency group with the prefix,
  Keep2Snapshot_KP_TEST_1005_1001_, and delete them.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SPECTRUM COPY D

• Fixed component ID

  5737B3400

Applicable component levels