4040 Replication suspended and a full resynchronization is required for one or more volume groups.

Explanation

Replication between the production and the recovery system is suspended. The suspended state occurs when errors exist in the replication configuration or replication is stopped intentionally. A full resynchronization of one or more volume groups is required.

User response

This event is logged when a system that uses policy-based replication is recovered after a severe system outage. After recovery, the system suspends replication to maintain the existing recovery point of replicated volume groups and prevent potential data loss from being replicated.

When replication is restarted after a system outage, a full resynchronization of the copies is required. The resynchronization process temporarily uses more capacity to maintain the current recovery point until a full resynchronization completes. Replication must be restarted for each volume group individually. You can choose to limit the number of volume groups that are resynchronizing at one time to manage the additional capacity required during the synchronization. Before you restart resynchronization, prioritize the order of the volume groups, monitor system capacity, and provide additional capacity, if necessary.

The event is automatically marked as fixed when no more volume groups with suspended status are on the system.

Identify the replication topology in use and follow the appropriate procedure:

  • Policy-based async replication
  • Policy-based high availability (PBR-HA)
Policy-based async replication
To unsuspend replication and restart replication, you need to determine which system has replication status of suspended. Enter the following command on one of the systems defined in the replication policy:
lsvolumegroupreplication <volumegroup_id/name>
Where volumegroup_id or name is the ID or name of the volume group. You can run this command on either the production or recovery system to display the status of both locations. For example:
local_location 1
location1_system_name system1
location1_replication_mode production
location1_volumegroup_id 1
location1_status suspended

In this case, volume group 1 is reporting the suspended status on the production system.

If the recovery copy is suspended, the lsvolumegroupreplication command displays the following results:


local_location 2
location2_system_name system2
location2_replication_mode recovery
location2_volumegroup_id 5
location2_status suspended

In this case, volume group 5 on the recovery system is suspended.

Depending on the system that displays the suspended status, different user actions are required. Use the following information to resynchronize data:

If the recovery copy is suspended:
If the recovery copy is suspended, issue the following command on the recovery system:
chvolumegroupreplication -unsuspend <volumegroup_id/name>

This starts a resynchronization of the specified volume group. If you have multiple volume groups that need resynchronization, you can choose to limit the number of volume groups that are resynchronizing at one time to manage the additional capacity required during the synchronization.

If the production copy is suspended:

If the production copy is suspended, it indicates either a planned or unplanned outage on the production location. Depending on when the outage occurred and state of the data, you might have data loss if the outage is greater than of your Recovery Point Objective (RPO).

After the outage is resolved, verify whether the data on the production volumes is consistent and not corrupted.

If the data is consistent, enter the following command on the production system:
chvolumegroupreplication -unsuspend <volumegroup_id/name>

If the data is not consistent, complete the following step:

For volume groups where replication is managed using external software:

If the replication for a volume group is managed by external orchestration software, such as VMware Site Recovery Manager (SRM), use the appropriate workflow in that application to failover to the recovery copy.

Verify that the data is usable on the recovery volume groups.
Note: If data is inconsistent with the production copy, determine whether the data loss is acceptable for your current RPO.
Run the following command on the production system to allow replication to be restarted:
chvolumegroupreplication -unsuspended <volume group ID | name>

Use the appropriate application workflow to restart replication.

After the data is resynchronized on the original production system, you can change the direction of the replication back to the original configuration by using the appropriate application workflow.

For volume groups where replication is managed using the native storage system:
On the recovery system, enter the following command:
chvolumegroupreplication -mode independent <volumegroup_id/name>

This command fails over to the recovery system and the recovery volume groups. Hosts are able to access the volumes while the volume group is in independent mode.

Verify that the data is usable on the recovery volume groups.
Note: If data is inconsistent with the production copy, determine whether the data loss is acceptable for your current RPO.

If you are satisfied with the data, run the following command on the production system:

chvolumegroupreplication -unsuspended <volume group ID | name>

Run the following command on the recovery system to restart replication using this copy as the production copy:

chvolumegroupreplication -mode production <volume group ID | name>
This command makes the recovery system the new production system. The data is replicated back to the original production system.
After the data is resynchronized on the original production system, you can change the direction of the replication back to the original configuration by enabling access to the recovery copy on the original production system and then restarting replication with that copy as production.  These actions can be performed by using the management GUI on the original production system, or by entering the following commands on the original production system:
chvolumegroupreplication -mode independent <volume group ID | name>
chvolumegroupreplication -mode production <volume group ID | name>
Policy-based high availability (PBR-HA)
Note: In a policy-based high availability (PBR-HA) configuration, workloads are managed at the storage partition level. Each storage partition has an Active Management System (AMS) — the system that owns the partition and from which all management commands must be issued — and a Preferred Management System (PMS) — the system designated by the administrator to act as AMS under normal conditions. After a system recovery, the partition configuration, AMS role, PMS designation, and replication policy association are all preserved, but replication across the affected partitions is suspended and a full resynchronization is required.

Unlike async policy-based replication, the recovery procedure for PBR-HA is driven at the partition level, not at the individual volume group level. You check partition health, confirm the AMS, and unsuspend at the partition level. Once unsuspended, all volume groups within the partition resynchronize automatically from the surviving system — no mode-change commands are needed or permitted.

Step 1: Check all partition statuses from the recovered system
On the recovered system, run the following command to see the health of every storage partition and identify which partitions have suspended replication: 
lspartition

Review the ha_status and per-location status fields for each partition. A partition that needs attention shows ha_status suspended with one or both location statuses also showing suspended.

Example output showing a partition with suspended replication on the recovered system: 
id  name        preferred_management_system_name  active_management_system_name  replication_policy_name  location1_system_name  location1_status  location2_system_name  location2_status  ha_status    link_status
0   partition0  site1system                       site1system                    HaRepPolicy              site1system            suspended         site2system            healthy           suspended    suspended

In this example, partition0 on site1system (the recovered system) is suspended. The surviving system (site2system) remains healthy. The AMS for this partition is site1system — all subsequent commands must be run on that system.

Note: If any partition shows ha_status problem, contact your IBM Support Center for further assistance before proceeding.
Step 2: Confirm you are on the AMS for each suspended partition

All recovery commands must be issued from the Active Management System (AMS) of the partition. The active_management_system_name field in the lspartition output identifies the AMS for each partition.

Important: If the AMS shown is not the system you have recovered, or if the AMS is not currently accessible, contact your IBM Support Center before proceeding.
Step 3: Verify partition data consistency

Before resuming replication, confirm that the data on the recovered partition is consistent. Because the recovered system is the AMS, I/O was being served from this system at the time of the outage. The surviving system held the synchronized Non-AMS copy.

  • If the data on the recovered system is consistent, proceed to Step 4.
  • If the data on the recovered system is not consistent, contact your IBM Support Center before proceeding.
CAUTION:
Do not attempt to unsuspend replication without guidance from IBM Support if the data on the recovered system is inconsistent. Unsuspending an inconsistent AMS copy risks overwriting the valid data held on the surviving system.
Step 4: Unsuspend replication for each suspended partition
For each partition that shows a suspended status, unsuspend replication from the AMS. Because the replication policy is an HA policy, there is no single partition-level unsuspend command — you unsuspend at the partition level by running the following command once for each volume group that the partition contains: 
chvolumegroupreplication -unsuspend <volumegroup_id/name>

To list the volume groups in a partition, run lsvolumegroup on the AMS and filter by the partition_name field that matches the suspended partition.

Once unsuspended, the partition automatically drives a full resynchronization of all its volume groups from the surviving system back to the recovered system. The AMS and PMS roles, replication policy, and partition configuration remain unchanged — you do not need to issue any mode-change commands.

To manage the capacity required during resynchronization, stagger the unsuspend operations across partitions — complete one partition before starting the next, or limit the number of volume groups resynchronizing in parallel across partitions.

Important: Do not run chvolumegroupreplication -mode production or chvolumegroupreplication -mode independent on any partition in a PBR-HA configuration. These mode-change commands are not supported for HA replication policies and will fail with an error. The partition manages replication direction and AMS/PMS failover automatically.
Step 5: Monitor recovery progress at the partition level
After unsuspending, track recovery progress by monitoring each partition rather than individual volume groups. Run the following command on the AMS: 
lspartition <partition_id/name>

Key fields to monitor:

  • ha_status — returns to established when the partition is fully synchronized and healthy.
  • location1_status / location2_status — both return to healthy when replication is running normally for all volume groups in the partition.
  • link_status — returns to active when the inter-system link is operating normally.

The 4040 event is automatically marked as fixed when no volume groups with suspended status remain on the system.

You can also monitor partition replication status from the management GUI by navigating to the Storage partitions page and checking the replication status for each partition.