Failback with migrated file on old primary and update contents from acting primary

This use case describes failing back with migrated file on old primary.

Fileset: primary
State of the file in latest RPO at primary: resident
Action on the file from primary after latest RPO: Change ctime and migrate
Primary disaster, failover to secondary, Write to file on acting primary, old primary came back up, run failback start
State of the file in the old primary after failback start: Migrated with no blocks
Current state of the file on acting primary: R with no mtime change
State of the file in the old primary after applyUpdates: migrated with no blocks

#create a new file from primary, flush to secondary

hs22n19; Thu Sep 24 08:25:53; tests# mmafmctl fs2 getstate -j drHSM-DRP

Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec 
------------ -------------- ------------- ------------ ------------ ------------- 
drHSM-DRP nfs://hs21n30/gpfs/fs1/drHSM-DRS Active hs22n19 0 103

#Change ctime and migrate on Primary after RPO snapshot

0 ;) hs22n19; Thu Sep 24 08:36:50; tests# chmod 333 /gpfs/fs2/drHSM-DRP/file19

0 ;) hs22n19; Thu Sep 24 08:36:54; tests# mmafmctl fs2 getstate -j drHSM-DRP

Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec 
------------ -------------- ------------- ------------ ------------ ------------- 
drHSM-DRP nfs://hs21n30/gpfs/fs1/drHSM-DRS Dirty hs22n19 9 104

hs22n21; Thu Sep 24 08:37:21; scripts# dsmmigrate /gpfs/fs2/drHSM-DRP/file19


ANS0102W Unable to open the message repository /opt/tivoli/tsm/client/ba/bin/EN_IN/dsmclientV3.cat. The American English repository will be used instead.
Tivoli Storage Manager
Command Line Space Management Client Interface
Client Version 7, Release 1, Level 3.0
Client date/time: 09/24/2015 08:37:41 EDTA
(c) Copyright by IBM Corporation and other(s) 1990, 2015. All Rights Reserved.
0 ;) hs22n21; Thu Sep 24 08:37:46; scripts# dsmls /gpfs/fs2/drHSM-DRP/file19 | grep file19

1048576 0 0 m file19
hs21n30; Thu Sep 24 08:39:00; drHSM-DRS# dsmls file19 | grep file19

1048576 1048576 1024 r file19

#Run failoverToSecondary at secondary

hs21n30; Thu Sep 24 08:39:32; fs1# mmafmctl fs1 failoverToSecondary -j drHSM-DRS --restore

mmafmctl: failoverToSecondary restoring from psnap failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47
[2015-09-24 08:39:34] Restoring fileset "drHSM-DRS" from snapshot "failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47" of filesystem "/dev/fs1"

[2015-09-24 08:39:41] Scanning inodes, phase 1 ...
[2015-09-24 08:39:44] 5777216 inodes have been scanned, 50% of total.
[2015-09-24 08:39:46] 11554432 inodes have been scanned, 100% of total.
[2015-09-24 08:39:46] Constructing operation list, phase 2 ...
[2015-09-24 08:39:46] 0 operations have been added to list.
[2015-09-24 08:39:46] 10 operations have been added to list.
[2015-09-24 08:39:46] Deleting the newly created files, phase 3 ...
[2015-09-24 08:39:47] Deleting the newly created hard links, phase 4 ...
[2015-09-24 08:39:47] Splitting clone files, phase 5 ...
[2015-09-24 08:39:48] Deleting the newly created clone files, phase 6 ...
[2015-09-24 08:39:49] Moving files, phase 7 ...
[2015-09-24 08:39:49] Reconstructing directory tree, phase 8 ...
[2015-09-24 08:39:50] Moving files back to their correct positions, phase 9 ...
[2015-09-24 08:39:51] Re-creating the deleted files, phase 10 ...
[2015-09-24 08:39:51] Re-creating the deleted clone parent files, phase 11 ...
[2015-09-24 08:39:52] Re-creating the deleted clone child files, phase 12 ...
[2015-09-24 08:39:53] Re-creating the deleted hard links, phase 13 ...
[2015-09-24 08:39:53] Restoring the deltas of changed files, phase 14 ...
[2015-09-24 08:39:54] Restoring the attributes of files, phase 15 ...
[2015-09-24 08:39:55] Restore completed successfully.
[2015-09-24 08:39:55] Clean up.
Primary Id (afmPrimaryID) 12646758592946767367-C0A874195583B621-19
Fileset drHSM-DRS changed.
Promoted fileset drHSM-DRS to Primary

0 ;) hs21n30; Thu Sep 24 08:39:55; fs1# cd /gpfs/fs1/drHSM-DRS

0 ;) hs21n30; Thu Sep 24 08:40:02; drHSM-DRS# dsmls file19 | grep file19 

1048576 1048576 1024 r file19

# Start failbackToPrimary at old primary

hs22n19; Thu Sep 24 08:40:13; tests# mmafmctl fs2 failbackToPrimary -j drHSM-DRP --start

Fileset drHSM-DRP changed.
mmafmctl: failbackToPrimary restoring from psnap failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47
[2015-09-24 08:40:18] Restoring fileset "drHSM-DRP" from snapshot "failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47" of filesystem "/dev/fs2"

[2015-09-24 08:40:20] Scanning inodes, phase 1 ...
[2015-09-24 08:40:23] 9447232 inodes have been scanned, 50% of total.
[2015-09-24 08:40:26] 18894464 inodes have been scanned, 100% of total.
[2015-09-24 08:40:26] Constructing operation list, phase 2 ...
[2015-09-24 08:40:26] 0 operations have been added to list.
[2015-09-24 08:40:27] 10 operations have been added to list.
[2015-09-24 08:40:27] Deleting the newly created files, phase 3 ...
[2015-09-24 08:40:27] Deleting the newly created hard links, phase 4 ...
[2015-09-24 08:40:28] Splitting clone files, phase 5 ...
[2015-09-24 08:40:29] Deleting the newly created clone files, phase 6 ...
[2015-09-24 08:40:29] Moving files, phase 7 ...
[2015-09-24 08:40:31] Reconstructing directory tree, phase 8 ...
[2015-09-24 08:40:31] Moving files back to their correct positions, phase 9 ...
[2015-09-24 08:40:32] Re-creating the deleted files, phase 10 ...
[2015-09-24 08:40:33] Re-creating the deleted clone parent files, phase 11 ...
[2015-09-24 08:40:33] Re-creating the deleted clone child files, phase 12 ...
[2015-09-24 08:40:34] Re-creating the deleted hard links, phase 13 ...
[2015-09-24 08:40:34] Restoring the deltas of changed files, phase 14 ...
[2015-09-24 08:40:35] Restoring the attributes of files, phase 15 ...
[2015-09-24 08:40:37] Restore completed successfully.
[2015-09-24 08:40:37] Clean up.

#State of the file in the old primary after failback start

hs22n21; Thu Sep 24 08:38:45; scripts# dsmls /gpfs/fs2/drHSM-DRP/file19 | grep file19 

1048576 0 0 m file19

#Write to file on acting primary to make it resident

hs21n30; Thu Sep 24 08:41:22; drHSM-DRS# dd if=/dev/urandom of=file19 bs=1M seek=2 count=1

1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.155252 s, 6.8 MB/s
0 ;) hs21n30; Thu Sep 24 08:42:31; drHSM-DRS# dsmls file19 | grep file19 

3145728 3145728 1024 r file19

#ApplyUpdates on old primary

hs22n19; Thu Sep 24 08:40:50; tests# mmafmctl fs2 applyUpdates -j drHSM-DRP

[2015-09-24 08:43:33] Getting the list of updates from the acting Primary...
[2015-09-24 08:43:48] Applying the 16 updates...
[2015-09-24 08:43:51] 16 updates have been applied, 100% of total.
mmafmctl: Creating the failback psnap locally. failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-43-32
Flushing dirty data for snapshot drHSM-DRP::failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-43-32...
Quiescing all file system operations.
Snapshot drHSM-DRP::failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-43-32 created with id 9020.
mmafmctl: Deleting the old failback psnap. failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47
Invalidating snapshot files in drHSM-DRP::failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47...
Deleting files in snapshot drHSM-DRP::failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47...
97.45 % complete on Thu Sep 24 08:46:23 2015 ( 9792 inodes with total 0 MB data processed)
100.00 % complete on Thu Sep 24 08:46:25 2015 ( 10048 inodes with total 0 MB data processed)
Invalidating snapshot files in drHSM-DRP::failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47/F/...
Delete snapshot drHSM-DRP::failback-psnap-rpo-C0A874465450C18C-46-15-09-24-08-16-47 complete, err = 0
hs22n21; Thu Sep 24 08:41:32; scripts# dsmls /gpfs/fs2/drHSM-DRP/file19 | grep file19 ; done

     3145728      3145728         3072   m     file19