Recovery scenario using incremental resynchronization in a Metro/Global Mirror configuration

Edit online

Use this process to restart recovery using the incremental resynchronization function during an outage at the intermediate site.

About this task

In a Metro/Global Mirror configuration, if you lose access to the storage unit at the intermediate site (either in a planned or unplanned outage), you can restart a two-site Global Mirror environment between the local and remote sites. You can use the incremental resynchronization function to avoid having to run a full copy of the volumes from the local site to the remote site.

This scenario describes the steps for restarting the recovery environment running Global Mirror from the local site to the remote site using the incremental resynchronization function. For best management practices, combine the functions of a Metro/Global Mirror environment with automation such as Geographically Dispersed Parallel Sysplex (GDPS) to ensure continuous or near-continuous availability during outages, including disasters.
Notes:
  • The following assumptions are made before you initiate the steps in this scenario:
    • You have established all your Remote Mirror and Copy paths before you establish your pairs or initiate any of the incremental resynchronization process. If the paths are not established first, an error condition might result.
    • You have established your Metro Mirror volume pairs to use the incremental resynchronization function on each of the primary volumes when you configured your Metro/Global Mirror configuration.
    • You have specified the -mode full parameter for each of these volume pairs.
Notes:
  • The command parameters and options that are used in this scenario are examples.
  • Some of the query output is presented in table format for clarity. The actual report is not displayed in this format.
  • The output for some commands differs depending on the storage unit from which you issue the command.

Complete these steps for the recovery operation:

Procedure

  1. Enable the incremental resynchronization option for the A to B Metro Mirror volume pairs.
    If this is the first attempt to establish the volume pairs, specify -mode full as shown in the mkpprc command example. Otherwise, specify -mode nocp.
    Enter the mkpprc command at the dscli command prompt with the following parameters and variables: 
    dscli> mkpprc -dev IBM.2107-130165X
-remotedev IBM.2107-75ALA2P -type mmir 
-mode full -incrementalresync enable 2100-2107:2100-2107
    The following example represents the output: 
    CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship
2100:2100 successfully created.
CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 
2101:2101 successfully created.
    See Creating a Metro Mirror relationship for more information.
  2. Pause (suspend) all A to B Metro Mirror volume pairs.
    Some (but not all) volume pairs might have been suspended with the outage of the intermediate site.
    Note: If the consistency group function is being used, the automation application (such as GPDS) issued the freezepprc command and all devices are suspended.
    Enter the pausepprc command at the dscli command prompt with the following parameters and variables: 
    dscli> pausepprc -dev IBM.2107-130165X
-remotedev IBM.2107-75ALA2P 
-unconditional -at src 2100-2107

    The following example represents the output: 

    CMUC00157I pausepprc: Remote Mirror and Copy volume pair 2100 
relationship successfully paused.
CMUC00157I pausepprc: Remote Mirror and Copy volume pair 2101 
relationship successfully paused.

    See Pausing a Metro Mirror relationship for more information.

    Notes:
    1. With the volume pairs suspended, updates to the A volumes are marked in the change recording and out-of-synchronization bitmaps on the Metro Mirror A volumes at the local site.
    2. The master storage unit might have been in the process of using FlashCopy to copy the consistency group to the D volumes when the outage occurred and the consistency group formation was not able to complete. If so, you must verify the consistency group formation. See Querying Global Mirror processing for more information.
  3. Issue a failover command to the C to B volumes at the remote site, specifying the -cascade option: With the loss of the B volumes at the intermediate site, the state of the C volumes is changed from secondary duplex pending (or suspended) to Suspended Host Source when the command processes. Updates are collected in out-of-sync bitmaps.

    Enter the failoverpprc command at the dscli command prompt with the following parameters and variables: 

    dscli> failoverpprc -dev IBM.2107-1831760
-remotedev IBM.2107-75ALA2P 
-type gcp -cascade  2100-2107:2100-2107

    The following example represents the output: 

    CMUC00196I failoverpprc: Remote Mirror and Copy pair 2100:2100 
successfully reversed.
CMUC00196I failoverpprc: Remote Mirror and Copy pair 2101:2101 
successfully reversed.

    See Running a recovery failover operation for more information.

  4. After the failover operation, you can view the status of the volumes to determine the state of the volumes:
    From the remote site, enter the lspprc command at the dscli command prompt with the following parameters and variables: 
    dscli> lspprc -l 2100-2107
    The following example represents the output:
    ID State Reason Type
    Out of
    Sync
    Tracks
    Tgt
    Read
    Src
    Cascade
    2100:2100
    Suspended
    Host
    Source
    		 - Global Copy 0 Disabled Enabled
    2101:2101
    Suspended
    Host
    Source
    		 - Global Copy 0 Disabled Enabled
    Tgt
    Cas
    cade
    Date
    Sus-
    pended
    Source
    LSS
    Time-
    out
    (secs)
    Crit
    Mode
    First
    Pass
    Status
    Incre-
    mental
    Resync
    Tgt
    Write
    Invalid - 21 300 Disabled True Disabled Disabled
    Invalid - 21 300 Disabled True Disabled Disabled
  5. Attempt to clean up any surviving components of Global Mirror at the intermediate site, if needed.
    1. End the Global Mirror session at the master storage unit.
      Enter the rmgmir command at the dscli command prompt with the following parameters and variables (from the intermediate site): 
      dscli> rmgmir -dev IBM.2107-75ALA2P
-quiet -lss 20 -session 31
      The following example represents the output:
      CMUC00165I rmgmir: Global Mirror for session 31 successfully 
stopped.
    2. End the Global Mirror session at the subordinate storage units.
      Reissue the command if the Global Mirror session does not stop because of subordinate storage units still associated to the master storage unit. See Ending a Global Mirror session for more information.
  6. Verify the Global Mirror consistency group formation:
    If the intermediate site outage occurred in the middle of consistency group formation, you must determine whether the FlashCopy operations must be committed or reverted.
    Enter the lsflash command at the dscli command prompt with the following parameters and variables.
    dscli> lsflash -l 2100-2107
    See Viewing information about FlashCopy relationships for more information.
    The following table represents an example of the output:
    ID SrcLSS
    Seq-
    uence
    Num
    		 Timeout
    Active
    Copy
    		 Recording Persistent
    2100:2300 21 44357D55 300 Disabled Enabled Enabled
    2101:2301 21 44357D55 300 Disabled Enabled Enabled
    Revertible
    Source-
    Write
    Enabled
    Target-
    Write
    Enabled
    Back-
    ground
    Copy
    Out Of
    Sync
    Tracks
    Date
    Created
    Date-
    Synced
    Disabled Enabled Disabled Disabled 1525879
    Fri Mar 24
    09:45:54
    MST 2006
    Thu Apr
    06
    13:42:58
    MST 2006
    Disabled Enabled Disabled Disabled 1525879
    Fri Mar 24
    09:45:54
    MST 2006
    Thu Apr
    06
    13:42:58
    MST 2006
  7. Establish Global Copy relationships using the A and C volume pairs with the Incremental Resynchronization recover option:
    Enter the mkpprc command at the dscli command prompt with the following parameters and variables:
    dscli> mkpprc -dev IBM.
2107-130165X -remotedev IBM.
2107-1831760 -type gcp -incrementalresync recover 2100-2107:2100-2107
    See Creating a Global Copy relationship for more information.
    The following example represents the output:
    CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 
2100:2100 successfully created.
CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 
2101:2101 successfully created.
    Notes:
    1. The C volumes were primary suspended volumes that had Global Copy relationships with the B volumes, which were in Metro Mirror relationships with the A volumes.
    2. The Incremental Resynchronization function that is running on the A volumes is stopped. The tracks of data in the change recording and out-of-synchronization bitmaps are merged and copied from the A volumes to the C volumes.
  8. Wait for the first pass of Global Copy processing to complete between the A to C volumes:
    You can monitor this activity by querying the status of the volumes.

    From the local site, enter the lspprc command at the dscli command prompt with the following parameters and variables: 

    dscli>  lspprc –dev IBM.
2107-130165X –remotedev IBM.
2107-1831760 2101:2101 2100:2101
    The following example represents the output:
    ID State Reason Type
    Source-
    LSS
    Time-
    out
    (secs)
    Critical
    Mode
    First
    Pass
    Status
    IBM.2107-
    130165X
    /2101:
    IBM.2107-
    1831760
    /2101
    Copy
    Pend-
    ing
    		 -
    Global
    Copy
    IBM.2107-
    1831760
    /20
    		 300 Disabled True
    IBM.2107-
    130165X
    /2100:
    IBM.2107-
    1831760
    /2100
    Copy
    Pend-
    ing
    		 -
    Global
    Copy
    IBM.2107-
    1831760
    /20
    		 300 Disabled True
  9. When the first pass of Global Copy processing is completed, start the Global Mirror session on the A volumes.

    The master storage unit begins forming consistency groups for the specified Global Mirror session. Global Mirror runs from the local site to the remote site until the intermediate site is ready to resume operation.

    Enter the mkgmir command at the dscli command prompt with the following parameters and variables (from the local site): 
    mkgmir -dev IBM.2107-130165X
 -lss 07 -session 31
    The following example represents the output:
    CMUC00162I mkgmir: Global Mirror for session 31 successfully
 started.
    See Starting Global Mirror processing for more information.
    When the intermediate site has been recovered, the volumes at the intermediate site must be resynchronized with the local volumes.

    During the outage, data was written to the volumes at the local site. After the intermediate site is recovered, the volumes at the intermediate site must be resynchronized.

    The former Metro/Global Mirror configuration must be "cleaned up" to reestablish it back to its original configuration. A host connection to the storage unit at the intermediate site is required.

  10. Complete the following steps in preparation for a failback operation from the remote site to the intermediate site:
    1. End the Metro Mirror relationship between the A to B volumes at the intermediate site.
      Enter the rmpprc command at the dscli command prompt with the following parameters and variables: 
      dscli> rmpprc -quiet -dev IBM.2107-75ALA2P  -remotedev IBM.
2107-130165X -unconditional -at tgt 2100-2107
      The following example represents the output:
      CMUC00155I rmpprc: Remote Mirror and Copy volume pair :2100 relationship 
successfully withdrawn.
CMUC00155I rmpprc: Remote Mirror and Copy volume pair :2101 relationship 
successfully withdrawn.
      See Deleting a Metro Mirror relationship for more information.
    2. Pause (suspend) the B to C volume pairs if they are not already suspended. You can query the status of the volumes for this determination.
      From the remote site, enter the lspprc command at the dscli command prompt with the following parameters and variables: 
      dscli> lspprc –dev IBM.
2107-130165X –remotedev IBM.
2107-1831760 2101:2101 2100:2101

      The following example represents the output: See Pausing a Metro Mirror relationship for more information.

      ID State Reason Type
      Source-
      LSS
      Time-
      out
      (secs)
      Critical
      Mode
      First
      Pass
      Status
      IBM.2107-
      1831760
      /2100:
      IBM.2107-
      75ALA2P
      /2100
      Suspended
      Host
      Source
      		 - Global Copy
      IBM.2107-
      75ALA2P
      /21
      		 unknown Disabled True
      IBM.2107-
      1831760

      2101:
      IBM.2107-
      75ALA2P
      /2101
      Copy
      Pending
      		 - Global Copy
      IBM.2107-
      1831760
      /21
      		 300 Disabled True
      If necessary, clean up the former Global Mirror configuration at the intermediate site using the following two steps:
    3. End the Global Mirror session from the master storage unit at the intermediate site.
      Note: If the Global Mirror session was successfully stopped at the time of the outage, this step might not be necessary and it might generate an error message when the command processes.
      Enter the rmgmir command at the dscli command prompt with the following parameters and variables (from the intermediate site): 
      dscli> rmgmir -dev IBM.2107-1301261 -quiet  -lss 20 -session 31
      The following example represents the output:
      CMUC00165I rmgmir: Global Mirror for session 31 successfully stopped.

      See Ending Global Mirror processing (script mode) or Ending Global Mirror processing (no script) for more information.

    4. If required, stop the Global Mirror session that is running from any of the subordinates.
      Enter the rmgmir command at the dscli command prompt with the following parameters and variables: 
      dscli> rmgmir -quiet  -lss 20 -session 31
      The following example represents the output:
      CMUC00165I rmgmir: Global Mirror for session 31 successfully 
stopped.

      See Ending Global Mirror processing (script mode) or Ending Global Mirror processing (no script) for more information.

  11. From the remote site, a failback Global Copy operation between the C to B volumes:
    When the failbackpprc command processes, data will be copied from the remote site to the intermediate site. Specify the C volumes as the sources and the B volumes as targets with the failback command.
    Note: Ensure the availability of the paths from the remote site to the intermediate site with the lspprcpath command.
    Enter the failbackpprc command at the dscli command prompt with the following parameters and variables: 
    dscli> failbackpprc -dev IBM.
2107-1831760 -remotedev IBM.
2107-75ALA2P -type gcp -cascade 2100-2107:2100-2107
    The following example represents the output:
    CMUC00197I failbackpprc: Remote Mirror and Copy pair 1A00:1200 successfully
 failed back. 
CMUC00197I failbackpprc: Remote Mirror and Copy pair 1A01:1201 successfully 
failed back.
    See Running a recovery failback operation for more information.
  12. Wait for the first pass to complete between the C volumes at the remote site and the B volumes at the intermediate site:
    You can monitor this activity by querying the status of the volumes.
    Enter the lspprc command at the dscli command prompt with the following parameters and variables: (from the intermediate site)
    dscli> lspprc -dev  IBM.
2107-1831760 -remotedev IBM.
2107-75ALA2P -l -fullid -fmt default 2100-2107
    See Querying Global Mirror processing for more information.
    The following example represents the output:
    ID State Reason Type
    Out of
    Sync
    Tracks
    Tgt
    Read
    Src
    Cascade
    IBM.2107-
    1831760
    /2100:
    IBM.2107-
    75ALA2P
    /2100
    Copy
    Pending
    		- Global Copy 0 Disabled Enabled
    IBM.2107-
    1831760
    /2101:
    IBM.2107-
    75ALA2P
    /2101
    Copy
    Pending
    		 - Global Copy 0 Disabled Enabled
    Tgt
    Cascade
    Date
    Suspended
    Source
    LSS
    Time-
    out
    (secs)
    Crit
    Mode
    First
    Pass
    Status
    Incre-
    mental
    Resync
    Tgt
    Write
    Invalid -
    IBM.2107-
    1831760
    /21
    		 Unknown Disabled True Disabled Disabled
    Invalid -
    IBM.2107-
    1831760
    /21
    		 Unknown Disabled True Disabled Disabled
  13. Start the Incremental Resynchronization function without the initialization option on the A volumes:
    This step allows you to "force" a resynchronization later between primary (A) volumes at the local site and the volumes at the intermediate site to ensure all updates are copied.
    Enter the mkpprc command at the dscli command prompt with the following parameters and variables: 
    dscli> mkpprc -dev IBM.2107-130165X -remotedev
IBM.2107-75ALA2P -type
gcp -incrementalresync enablenoinit -mode nocp 2100-2107:2100-2107
    The following example represents the output:
    CMUC00153I mkpprc: Remote Mirror and Copy volume pair 
relationship 2100:2100 successfully created.
CMUC00153I mkpprc: Remote Mirror and Copy volume pair 
relationship 2101:2101 successfully created.
    See Creating a Metro Mirror relationship for more information.
    You are now ready to restore the original configuration Metro/Global Mirror without interrupting production.
  14. Stop the Global Mirror session between the A and C volumes between the local and remote sites.
    During this transition time, the data on the D volumes in FlashCopy relationships might be consistent but not current until the transition is complete.
    Enter the rmgmir command at the dscli command prompt with the following parameters and variables: 
    dscli> rmgmir -dev IBM.2107-130165X -quiet  -lss 21 -session 31
    The following example represents the output:
    CMUC00165I rmgmir: Global Mirror for session 31 successfully stopped.

    See Ending Global Mirror processing (script mode) or Ending Global Mirror processing (no script) for more information.

  15. Allow the resynchronization of the C to B volumes to run by completing the following steps:
    1. Pause (suspend) the A to C volume pairs that were established in Global Copy mode.
      Enter the pausepprc command at the dscli command prompt with the following parameters and variables: 
      dscli> pausepprc -dev IBM.
2107-130165X -remotedev IBM.
2107-1831760 2100-2107:2100-2107
      The following example represents the output:
      CMUC00157I pausepprc: Remote Mirror and Copy volume pair 2100:2100
 relationship successfully paused.
CMUC00157I pausepprc: Remote Mirror and Copy volume pair 2101:2101 
relationship successfully paused.
      See Pausing a Metro Mirror relationship for more information.
    2. Wait for data to be copied from the C volumes at the remote site to the B volumes at the intermediate site.
      Enter the lspprc command at the dscli command prompt with the following parameters and variables: 
      dscli> lspprc -dev IBM.
2107-1831760 -remotedev IBM.
2107-75ALA2P -l -fmt default 2100-2107
      See Querying Global Mirror processing for more information.
      The following example represents the output:
      ID State Reason Type
      Out of
      Sync
      Tracks
      Tgt
      Read
      Src
      Cascade
      IBM.2107-
      1831760
      /2100:
      IBM.2107-
      75ALA2P
      /2100
      Copy
      Pending
      		- Global Copy 0 Disabled Enabled
      IBM.2107-
      1831760
      /2101:
      IBM.2107-
      75ALA2P
      /2101
      Copy
      Pending
      		 - Global Copy 0 Disabled Enabled
      Tgt
      Cascade
      Date
      Sus-
      pended
      Source
      LSS
      Time-
      out
      (secs)
      Crit
      Mode
      First
      Pass
      Status
      Incre-
      mental
      Resync
      Tgt
      Write
      Invalid -
      IBM.2107-
      1831760
      /21
      		 Unknown Disabled True Disabled Disabled
      Invalid -
      IBM.2107-
      1831760
      /21
      		 Unknown Disabled True Disabled Disabled

    3. End the A and C Global Copy relationship at the remote site.
      Enter the rmpprc command at the dscli command prompt with the following parameters and variables: 
      dscli> rmpprc -quiet -dev IBM.2107-1831760
-unconditional -at tgt 2100-2107
      The following example represents the output:
      CMUC00155I rmpprc: Remote Mirror and Copy volume pair :2100 
relationship successfully withdrawn.
CMUC00155I rmpprc: Remote Mirror and Copy volume pair :2101 
relationship successfully withdrawn.
      See Removing the Global Copy pair relationship for more information.
      Notes:
      1. The value for the -dev parameter must be the remote site server (site C).
      2. The management console must be able to communicate with the remote server for this command to process successfully.

      When the command processes, the C volumes at the remote site are no longer the secondary volumes in a Global Copy relationship with the A volumes. This process allows for a later failback operation for the B to C volume pairs.

      The Global Copy relationship between the A to C volumes was stopped at the remote site, which did not affect the status of the A volumes at the local site. The updates on the A volumes continue until the volumes are again fully synchronized.

  16. After data on the C volumes has been copied to the B volumes, pause (suspend) the C to B volume pairs.
    This step is required before a failback operation can be issued between the B to C volumes, which requires the C volumes to be paused.
    Enter the pausepprc command at the dscli command prompt with the following parameters and variables: 
    dscli> pausepprc -dev IBM.
2107-1831760 -remotedev IBM.
2107-75ALA2P 2100-2107:2100-2107
    The following example represents the output:
    CMUC00157I pausepprc: Remote Mirror and Copy volume pair 2100:2100 
relationship successfully paused.
CMUC00157I pausepprc: Remote Mirror and Copy volume pair 2101:2101 
relationship successfully paused.
    See Pausing a Metro Mirror relationship for more information.
  17. At the intermediate site, issue a failover Global Copy operation to the B to C volumes, with the -cascade option:
    The B volumes are primary suspended volumes.
    Enter the failoverpprc command at the dscli command prompt with the following parameters and variables: 
    dscli> failoverpprc -dev IBM.
2107-75ALA2P -remotedev IBM.
2107-1831760 -type gcp -cascade 2100-2107:2100-2107
    The following example represents the output:
    CMUC00196I failoverpprc: Remote Mirror and Copy pair 2100:2100 
successfully reversed.
CMUC00196I failoverpprc: Remote Mirror and Copy pair 2101:2101 
successfully reversed.
    See Running a recovery failover operation for more information.
  18. At the intermediate site, run a failback Global Copy operation for the B to C volumes, with the -cascade option:
    Enter the failbackpprc command at the dscli command prompt with the following parameters and variables: 
    dscli> failbackpprc -dev IBM.
2107-75ALA2P -remotedev IBM.
2107-1831760 -type gcp -cascade 2100-2107:2100-2107
    The following example represents the output:
    CMUC00197I failbackpprc: Remote Mirror and Copy pair 2100:2100 successfully 
failed back. 
CMUC00197I failbackpprc: Remote Mirror and Copy pair 2101:2101 successfully 
failed back.
    See Running a recovery failback operation for more information.
  19. Establish Metro Mirror relationships between the A to B volumes using the incremental resynchronization function and the override option.
    As a result, the relationship verification is bypassed and the incremental resynchronization function stopped. The change recording and out-of-synchronization bitmaps that were monitored and tracked on the primary Metro Mirror volumes are merged to determine the data to copy from the A to B Metro Mirror volume pairs. A full volume copy is bypassed and only changes are copied from the A volumes to the B volumes.
    Enter the mkpprc command at the dscli command prompt with the following parameters and variables: 
    dscli> mkpprc -dev IBM.2107-130165X -remotedev
IBM.2107-75ALA2P -type
mmir -mode nocp -incrementalresync override 2100-2107:2100-2107
    The following example represents the output:
    CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 2100:2100 
successfully created.
CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 2101:2101 
successfully created.
    See Creating a Metro Mirror relationship for more information.
  20. At local site, start the incremental resynchronization with the initialization option on the A volumes in Metro Mirror relationships.
    The first pass of copying data between the A to B volumes starts (without a full copy). The B to C volumes data copying can also be in the first pass resulting from the failback operation.
    Enter the mkpprc command at the dscli command prompt with the following parameters and variables: 
    dscli> mkpprc -dev IBM.2107-130165X -remotedev
IBM.2107-1301261 -type
mmi -mode nocp -incrementalresync enable 2100-2107:2100-2107
    The following example represents the output:
    CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 2100:2100 
successfully created.
CMUC00153I mkpprc: Remote Mirror and Copy volume pair relationship 2101:2101 
successfully created.
    See Creating a Metro Mirror relationship for more information.
  21. Wait until the first pass of the A to B volume pairs to reach full duplex:
    You can monitor this activity by querying the status of the A to B volumes. As soon as the number of out-of-sync tracks reaches zero, all data has been copied and the data on the A to B volumes is equal. Global Mirror processing starts to form consistency groups when the status of the A to B volumes is full duplex.

    See Viewing information about Metro Mirror relationships for more information.

  22. Start the Global Mirror session at the intermediate site:
    Enter the mkgmir command at the dscli command prompt with the following parameters and variables (from the local site): 
    mkgmir -dev IBM.2107-75ALA2P
-lss 07 -session 31
    The following example represents the output:
    CMUC00162I mkgmir: Global Mirror for session 31 successfully started.
    See Starting Global Mirror processing for more information.

Results

Your original configuration is restored.