Read a migrated file from acting primary after failover

This use case describes reading a migrated file from acting primary after failover.

Fileset: primary 
          State of the file in latest RPO at secondary: resident
          Action on the file from primary (synced to secondary): Change ctime of the file and flush request
          TSM action on the file from secondary: migrate the file
          State of the file in the live filesystem of the secondary: migrated
          Action: Unlink primary to feign disaster and failover to secondary
          State of the file in the live filesystem of the acting primary (after failover): migrated
          Action on the file from acting primary: Read the file
          State of the in the live filesystem on the acting primary: pre-migrated

#create a new file from primary, flush to secondary

hs22n21; Wed Sep 16 07:29:37; ~ mmafmctl fs2 getstate-j drHSM-DRP-5


Fileset Name    Fileset Target                                Cache State          Gateway Node    Queue Length   Queue numExec 
------------    --------------                                -------------        ------------    ------------   ------------- 
drHSM-DRP-5     nfs://hs21n30/gpfs/fs1/drHSM-DRS-5            Active               hs22n18         0              2

#create a new RPO snapshot from primary 0 ;) hs22n20; Wed Sep 16 07:30:12; ~ mmpsnap fs2 create -j drHSM-DRP-5 --rpo


Flushing dirty data for snapshot drHSM-DRP-5::psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17...
Quiescing all file system operations.
Snapshot drHSM-DRP-5::psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17 created with id 8153.
mmpsnap: The peer snapshot psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17 is created successfully.

list all RPO snapshots from primary

0 ;) hs22n20; Wed Sep 16 07:30:19; ~ mmlssnapshot fs2 -j drHSM-DRP-5

Snapshots in file system fs2:


Directory                SnapId    Status  Created                   Fileset
psnap0-rpo-C0A874465450C18C-132 8089      Valid   Wed Sep 16 05:16:54 2015  drHSM-DRP-5
psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17 8153      Valid   Wed Sep 16 07:30:18 2015  drHSM-DRP-5

0 ;) hs22n20; Wed Sep 16 07:30:22; ~ mmafmctl fs2 getstate-j drHSM-DRP-5

Fileset Name    Fileset Target                                Cache State          Gateway Node    Queue Length   Queue numExec 
------------    --------------                                -------------        ------------    ------------   ------------- 
drHSM-DRP-5     nfs://hs21n30/gpfs/fs1/drHSM-DRS-5            Active               hs22n18         0              15

#listing at secondary

hs21n30; Wed Sep 16 07:29:54; NewSanity ls -lashi /gpfs/fs1/drHSM-DRS-5


total 11M
12058627    0 drwxrwxrwx  3 root root 4.0K Sep 16 07:30 .
       3 256K drwxr-xr-x 91 root root 256K Sep 16 07:26 ..
12058628    0 drwxr-xr-x  3 root root 4.0K Sep 16 05:16 .afm
      39  512 dr-xr-xr-x  2 root root 8.0K Dec 31  1969 .snapshots
12058624  10M -rw-r--r--  1 root root  10M Sep 16 07:29 file

#listing RPO snapshot contents at secondary

0 ;) hs21n30; Wed Sep 16 07:30:58; NewSanity ls -lashi /gpfs/fs1/drHSM-DRS-5/.snapshots/psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17/


total 11M
     327   0 drwxrwxrwx 3 root root 4.0K Sep 16 07:30 .
      39 512 dr-xr-xr-x 2 root root 8.0K Dec 31  1969 ..
12058628   0 drwxr-xr-x 3 root root 4.0K Sep 16 05:16 .afm
12058624 10M -rw-r--r-- 1 root root  10M Sep 16 07:29 file

#checking the file on live filesystem and on last RPO snapshot at secondary

130 ;( hs21n30; Wed Sep 16 07:31:05; NewSanity md5sum /gpfs/fs1/drHSM-DRS-5/file


b48ba201b5b820d147e5d8b95fd96309  /gpfs/fs1/drHSM-DRS-5/file

#modify the primary after last RPO snapshot was triggered, and flush changes to secondary.

hs21n20; Wed Sep 16 07:32:00; NewSanity chmod 777 /gpfs/fs1/drHSM-DRP-5/file

#List the file in secondary live fs and on the last RPO snapshot

0 ;) hs21n30; Wed Sep 16 07:32:32; NewSanity ls -lashi /gpfs/fs1/drHSM-DRS-5


total 11M
12058627    0 drwxrwxrwx  3 root root 4.0K Sep 16 07:30 .
       3 256K drwxr-xr-x 91 root root 256K Sep 16 07:26 ..
12058628    0 drwxr-xr-x  3 root root 4.0K Sep 16 05:16 .afm
      39  512 dr-xr-xr-x  2 root root 8.0K Dec 31  1969 .snapshots
12058624  10M -rwxrwxrwx  1 root root  10M Sep 16 07:29 file

hs21n30; Wed Sep 16 07:33:05; NewSanity dsmls /gpfs/fs1/drHSM-DRS-5/file


Tivoli Storage Manager
Command Line Space Management Client Interface
  Client Version 7, Release 1, Level 3.0
  Client date/time: 09/16/15   07:33:08
(c) Copyright by IBM Corporation and other(s) 1990, 2015. All Rights Reserved.

	ActS         ResS         ResB   FSt    FName
    10485760     10485760        10240   r      file

#migrate the file on secondary live fs

0 ;) hs21n30; Wed Sep 16 07:33:08; NewSanity dsmmigrate /gpfs/fs1/drHSM-DRS-5/file


Tivoli Storage Manager
Command Line Space Management Client Interface
  Client Version 7, Release 1, Level 3.0
  Client date/time: 09/16/15   07:33:21
(c) Copyright by IBM Corporation and other(s) 1990, 2015. All Rights Reserved.

0 ;) hs21n30; Wed Sep 16 07:33:26; NewSanity dsmls /gpfs/fs1/drHSM-DRS-5/file


Tivoli Storage Manager
Command Line Space Management Client Interface
  Client Version 7, Release 1, Level 3.0
  Client date/time: 09/16/15   07:33:29
(c) Copyright by IBM Corporation and other(s) 1990, 2015. All Rights Reserved.

	ActS         ResS         ResB   FSt    FName
    10485760            0            0   m      file

#unlink primary to feign failure

hs22n20; Wed Sep 16 07:36:39; ~ mmunlinkfileset fs2 drHSM-DRP-5


Fileset drHSM-DRP-5 unlinked.

#failover to secondary

hs21n30; Wed Sep 16 07:33:44; NewSanity mmafmctl fs1 failoverToSecondary -j drHSM-DRS-5 --restore


mmafmctl: failoverToSecondary restoring from psnap psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17
[2015-09-16 07:33:47] Restoring fileset "drHSM-DRS-5" from snapshot "psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17" of filesystem "/dev/fs1"

[2015-09-16 07:33:54] Scanning inodes, phase 1 ...
[2015-09-16 07:33:54] 12059648 inodes have been scanned, 50% of total.
[2015-09-16 07:33:55] 24119296 inodes have been scanned, 100% of total.
[2015-09-16 07:33:55] Constructing operation list, phase 2 ...
[2015-09-16 07:33:55] 0 operations have been added to list.
[2015-09-16 07:33:55] 1 operations have been added to list.
[2015-09-16 07:33:55] Deleting the newly created files, phase 3 ...
[2015-09-16 07:33:55] Deleting the newly created hard links, phase 4 ...
[2015-09-16 07:33:56] Splitting clone files, phase 5 ...
[2015-09-16 07:33:57] Deleting the newly created clone files, phase 6 ...
[2015-09-16 07:33:58] Moving files, phase 7 ...
[2015-09-16 07:33:58] Reconstructing directory tree, phase 8 ...
[2015-09-16 07:33:59] Moving files back to their correct positions, phase 9 ...
[2015-09-16 07:34:00] Re-creating the deleted files, phase 10 ...
[2015-09-16 07:34:00] Re-creating the deleted clone parent files, phase 11 ...
[2015-09-16 07:34:01] Re-creating the deleted clone child files, phase 12 ...
[2015-09-16 07:34:02] Re-creating the deleted hard links, phase 13 ...
[2015-09-16 07:34:03] Restoring the deltas of changed files, phase 14 ...
[2015-09-16 07:34:03] Restoring the attributes of files, phase 15 ...
[2015-09-16 07:34:04] Restore completed successfully.
[2015-09-16 07:34:04] Clean up.
Primary Id (afmPrimaryID) 12646758592946767367-C0A874195583B621-54
Fileset drHSM-DRS-5 changed.
Promoted fileset drHSM-DRS-5 to Primary

#after failover, check status of the file on acting primary – live fs and in the last RPO snapshot

0 ;) hs21n30; Wed Sep 16 07:34:09; NewSanity dsmls /gpfs/fs1/drHSM-DRS-5/file


Tivoli Storage Manager
Command Line Space Management Client Interface
  Client Version 7, Release 1, Level 3.0
  Client date/time: 09/16/15   07:34:11
(c) Copyright by IBM Corporation and other(s) 1990, 2015. All Rights Reserved.

	ActS         ResS         ResB   FSt    FName
    10485760            0            0   m      file

0 ;) hs21n30; Wed Sep 16 07:34:11; NewSanity ls -lashi /gpfs/fs1/drHSM-DRS-5


total 257K
12058627    0 drwxrwxrwx  3 root root 4.0K Sep 16 07:30 .
       3 256K drwxr-xr-x 91 root root 256K Sep 16 07:34 ..
12058628    0 drwxr-xr-x  3 root root 4.0K Sep 16 05:16 .afm
      39  512 dr-xr-xr-x  2 root root 8.0K Dec 31  1969 .snapshots
12058624    0 -rw-r--r--  1 root root  10M Sep 16 07:29 file

0 ;) hs21n30; Wed Sep 16 07:34:14; NewSanity ls -lashi /gpfs/fs1/drHSM-DRS-5/.snapshots/psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17/


total 11M
     327   0 drwxrwxrwx 3 root root 4.0K Sep 16 07:30 .
      39 512 dr-xr-xr-x 2 root root 8.0K Dec 31  1969 ..
12058628   0 drwxr-xr-x 3 root root 4.0K Sep 16 05:16 .afm
12058624 10M -rw-r--r-- 1 root root  10M Sep 16 07:29 file

#read the file from acting primary

hs21n30; Wed Sep 16 07:34:56; NewSanity cat /gpfs/fs1/drHSM-DRS-5/file > /dev/null

#check file state again on acting primary

0 ;) hs21n30; Wed Sep 16 07:35:04; NewSanity dsmls /gpfs/fs1/drHSM-DRS-5/file


Tivoli Storage Manager
Command Line Space Management Client Interface
  Client Version 7, Release 1, Level 3.0
  Client date/time: 09/16/15   07:35:06
(c) Copyright by IBM Corporation and other(s) 1990, 2015. All Rights Reserved.

	ActS         ResS         ResB   FSt    FName
    10485760     10485760        10240   p      file

#check md5sum of the file on live fs and on RPO snapshot, in acting primary

hs21n30; Wed Sep 16 07:43:21; NewSanity md5sum /gpfs/fs1/drHSM-DRS-5/.snapshots/psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17/file


b48ba201b5b820d147e5d8b95fd96309  /gpfs/fs1/drHSM-DRS-5/.snapshots/psnap-rpo-C0A874465450C18C-132-15-09-16-07-30-17/file

0 ;) hs21n30; Wed Sep 16 07:43:56; NewSanity md5sum /gpfs/fs1/drHSM-DRS-5/file


b48ba201b5b820d147e5d8b95fd96309  /gpfs/fs1/drHSM-DRS-5/file