Convert GPFS-independent filesets to AFM DR, failover to secondary and failback to a new primary by using the outband trucking method

This use case scenario describes conversion of GPFS-independent filesets to primary-secondary using inband option, failover and failback to a new primary by using the outband trucking option.

For this scenario, the secondary's file system must be remote mounted via cross-cluster mount.

Note: Parallel data transfer is not functional during a scenario of failover and failback.

Create GPFS-independent fileset at primary site with contents:

mmcrfileset fs1 gpfs1022 --inode-space=new


    Fileset gpfs1022 created with id 185 root inode 49283075.
    mmlinkfileset fs1 gpfs1022 -J /gpfs/fs1/gpfs1022
    Fileset gpfs1022 linked at /gpfs/fs1/gpfs1022

/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/fs1/gpfs1022/file_gpfspri_1


      recSize 10K nBytes 100M fileSize 100M
      nProcesses 1 nThreadsPerProcess 1
      file cache flushed before test
      not using data shipping
      not using direct I/O
      offsets accessed will cycle through the same file segment
      not using shared memory buffer
      not releasing byte-range token after open
      no fsync at end of test
        Data rate was 73161.03 Kbytes/sec, thread utilization 0.996

/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/fs1/gpfs1022/file_gpfspri_2


      recSize 10K nBytes 100M fileSize 100M
      nProcesses 1 nThreadsPerProcess 1
      file cache flushed before test
      not using data shipping
      not using direct I/O
      offsets accessed will cycle through the same file segment
      not using shared memory buffer
      not releasing byte-range token after open
      no fsync at end of test
        Data rate was 73639.84 Kbytes/sec, thread utilization 1.000

    Contents on the GPFS fileset:

ls -l /gpfs/fs1/gpfs1022


    total 402944
    -rw-r--r-- 1 root root 104857600 Apr 28 03:52 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2

Create empty GPFS-independent fileset at secondary site:
/usr/lpp/mmfs/bin/mmcrfileset homefs1 gpfs1022 --inode-space=new
```
    Fileset gpfs1022 created with id 190 root inode 1345847299.
```
/usr/lpp/mmfs/bin/mmlinkfileset homefs1 gpfs1022 -J /gpfs/homefs1/gpfs1022
```
    Fileset gpfs1022 linked at /gpfs/homefs1/gpfs1022
```
ls -l /gpfs/homefs1/gpfs1022
```
    total 0
```
Note: After the GPFS fileset is linked, ownership and permissions are set to root and 0700 as default. If you want non-default permissions, modify now so that permissions of the root directory of this GPFS fileset are reflected at primary fileset after conversion.
Collect the primary Id of the primary fileset at primary site. An example of a command:
/usr/lpp/mmfs/bin/mmafmctl fs1 getPrimaryId -j gpfs1022

Primary Id (afmPrimaryID) 3235816757204815584-C0A802135530BD89-185

Note the Primary Id from the output. This is required during the further steps.
Convert the GPFS-independent fileset at secondary using above Primary Id:
/usr/lpp/mmfs/bin/mmchfileset homefs1 gpfs1022 -p afmmode=secondary -p afmPrimaryID=3235816757204815584-C0A802135530BD89-185

The following output is displayed: Fileset gpfs1022 changed.

Convert the GPFS fileset at primary site to primary using --inband option:

/usr/lpp/mmfs/bin/mmafmctl fs1 convertToPrimary -j gpfs1022 --afmtarget=gpfs:///gpfs/remotefs1/gpfs1022 -inband --afmRPO=720

The following output is displayed:

Checking for any special files. This may take a while...
Converting GPFS fileset to AFM primary fileset...
Primary Id (afmPrimaryID) 3235816757204815584-C0A802135530BD89-185
Fileset gpfs1022 changed.
Setting up a Primary and Secondary relation...
Data will be moved to secondary via AFM
psnap will be taken at secondary after data movement completes

Primary contents:

ls -l /gpfs/fs1/gpfs1022

total 409600
-rw-r--r-- 1 root root 104857600 Apr 28 03:52 file_gpfspri_1
-rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2

Secondary contents:

ls -l /gpfs/homefs1/gpfs1022

total 409600
-rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_1
-rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2

Create data from primary and ensure that it goes to secondary:

/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/fs1/gpfs1022/file_gpfspri_3


      recSize 10K nBytes 100M fileSize 100M
      nProcesses 1 nThreadsPerProcess 1
      file cache flushed before test
      not using data shipping
      not using direct I/O
      offsets accessed will cycle through the same file segment
      not using shared memory buffer
      not releasing byte-range token after open
      no fsync at end of test
        Data rate was 35139.11 Kbytes/sec, thread utilization 0.996

/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/fs1/gpfs1022/file_gpfspri_4


      recSize 10K nBytes 100M fileSize 100M
      nProcesses 1 nThreadsPerProcess 1
      file cache flushed before test
      not using data shipping
      not using direct I/O
      offsets accessed will cycle through the same file segment
      not using shared memory buffer
      not releasing byte-range token after open
      no fsync at end of test
        Data rate was 35568.11 Kbytes/sec, thread utilization 0.999

    Primary contents:

ls -l /gpfs/fs1/gpfs1022


    total 819200
    -rw-r--r-- 1 root root 104857600 Apr 28 03:52 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_4

    Secondary contents:

ls -l /gpfs/homefs1/gpfs1022


    total 614400
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_4

    Create user RPO:

mmpsnap fs1 create -j gpfs1022 --rpo


    Flushing dirty data for snapshot gpfs1022::psnap-rpo-C0A802135530BD89-185-15-04-28-03-54-37...
    Quiescing all file system operations.
    Snapshot gpfs1022::psnap-rpo-C0A802135530BD89-185-15-04-28-03-54-37 created with id 331.

Unlink primary feigning primary going down:
/usr/lpp/mmfs/bin/mmunlinkfileset fs1 gpfs1022 -f
```
        Fileset gpfs1022 unlinked.
```

Failover - convert secondary to acting primary:

/usr/lpp/mmfs/bin/mmafmctl homefs1 failoverToSecondary -j gpfs1022 --restore


    mmafmctl: failoverToSecondary restoring from psnap psnap-rpo-C0A802135530BD89-185-15-04-28-03-54-37
    [2015-04-28 03:54:40] Restoring fileset "gpfs1022" from snapshot "psnap-rpo-C0A802135530BD89-185-15-04-28-03-54-37" of
    filesystem "/dev/homefs1"
    [2015-04-28 03:54:42] Scanning inodes, phase 1 ...
    [2015-04-28 03:54:43] 1345947328 inodes have been scanned, 100% of total.
    [2015-04-28 03:54:43] There's no data changes since the restoring snapshot, skipping restore.
    [2015-04-28 03:54:43] Restore completed successfully.
    [2015-04-28 03:54:43] Clean up.
    Primary Id (afmPrimaryID) 2892567029667191230-7DA6C0A855122CF5-190
    Fileset gpfs1022 changed.
    Promoted fileset gpfs1022 to Primary

    Acting primary:

/usr/lpp/mmfs/bin/mmlsfileset homefs1 gpfs1022 -L -afm



    Filesets in file system 'homefs1':

    Attributes for fileset gpfs1022:
    =================================
    Status                                  Linked
    Path                                    /gpfs/homefs1/gpfs1022
    Id                                      190
    Root inode                              1345847299
    Parent Id                               0
    Created                                 Tue Apr 28 03:53:03 2015
    Comment
    Inode space                             87
    Maximum number of inodes                100032
    Allocated inodes                        100032
    Permission change flag                  chmodAndSetacl
    afm-associated                          Yes
    Target                                  nfs://(null)(null)
    Mode                                    primary
    Async Delay                             15 (default)
    Recovery Point Objective                disable
    Last pSnapId                            0
    Number of Gateway Flush Threads         4
    Primary Id                              2892567029667191230-7DA6C0A855122CF5-190

    Contents from acting primary after failover:

ls -l /gpfs/homefs1/gpfs1022


    total 614400
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_4

Create data from acting primary:

/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/homefs1/gpfs1022/file_actingpri_1


      recSize 10K nBytes 100M fileSize 100M
      nProcesses 1 nThreadsPerProcess 1
      file cache flushed before test
      not using direct I/O
      offsets accessed will cycle through the same file segment
      not using shared memory buffer
      not releasing byte-range token after open
      no fsync at end of test
        Data rate was 52699.54 Kbytes/sec, thread utilization 0.991

/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/homefs1/gpfs1022/file_actingpri_2


      recSize 10K nBytes 100M fileSize 100M
      nProcesses 1 nThreadsPerProcess 1
      file cache flushed before test
      not using direct I/O
      offsets accessed will cycle through the same file segment
      not using shared memory buffer
      not releasing byte-range token after open
      no fsync at end of test
        Data rate was 56026.22 Kbytes/sec, thread utilization 0.999

    Data from acting primary:

ls -l /gpfs/homefs1/gpfs1022/


    total 815104
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_actingpri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_actingpri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_4

A new primary is created.

Prepare a snapshot for new primary setup from acting primary. In these steps, a new GPFS independent fileset is created on another cluster, which needs to be converted as primary. A snapshot of the current acting primary is taken, from which data must be copied to this new fileset. After this the fileset can be converted to primary by using the outband trucking option. All these steps are as under:
/usr/lpp/mmfs/bin/mmafmctl homefs1 replacePrimary -j gpfs1022
```
      psnap0-newprimary-base-rpo-7DA6C0A855122CF5-190
```
Create new GPFS-independent fileset at new primary site:
mmcrfileset fs1 newgpfs1022 --inode-space=new
```
    Fileset newgpfs1022 created with id 187 root inode 49807363.
```
mmlinkfileset fs1 newgpfs1022 -J /gpfs/fs1/newgpfs1022
```
    Fileset newgpfs1022 linked at /gpfs/fs1/newgpfs1022
```

Copy/Truck contents from snapshot created in step 9, to a new primary by using outband trucking method:

psnap0-newprimary-base-rpo-7DA6C0A855122CF5-190


    New primary contents:

ls -l /gpfs/fs1/newgpfs1022


    total 1222656
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_4

    Create more data from acting primary in the meanwhile, to show that applications continue there:

/usr/lpp/mmfs/samples/perf/gpfsperf create seq /gpfs/homefs1/gpfs1022/file_actingpri_3


      recSize 10K nBytes 100M fileSize 100M
      nProcesses 1 nThreadsPerProcess 1
      file cache flushed before test
      not using direct I/O
      offsets accessed will cycle through the same file segment
      not using shared memory buffer
      not releasing byte-range token after open
      no fsync at end of test
        Data rate was 53044.68 Kbytes/sec, thread utilization 0.987

Convert new GPFS-independent fileset to primary by using the outband trucking method:

/usr/lpp/mmfs/bin/mmafmctl fs1 convertToPrimary -j newgpfs1022 --afmtarget gpfs:///gpfs/remotefs1/gpfs1022 --secondary-snapname psnap0-newprimary-base-rpo-7DA6C0A855122CF5-190 --inband


    
    Checking for any special files. This may take a while...
    Converting GPFS fileset to AFM primary fileset...
    Primary Id (afmPrimaryID) 3235816757204815584-C0A802135530BD89-187
    Fileset newgpfs1022 changed.
    Flushing dirty data for snapshot newgpfs1022::psnap0-newprimary-base-rpo-7DA6C0A855122CF5-190...
    Quiescing all file system operations.
    Snapshot newgpfs1022::psnap0-newprimary-base-rpo-7DA6C0A855122CF5-190 created with id 332.

    New primary contents:

ls -l /gpfs/fs1/newgpfs1022


    total 1228800
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_4

Run failback start on new primary:

/usr/lpp/mmfs/bin/mmafmctl fs1 failbackToPrimary -j newgpfs1022 --start


    Fileset newgpfs1022 changed.
    mmafmctl: failbackToPrimary restoring from psnap psnap0-newprimary-base-rpo-7DA6C0A855122CF5-190
    [2015-04-28 03:55:31] Restoring fileset "newgpfs1022" from snapshot "psnap0-newprimary-base-rpo-7DA6C0A855122CF5-190" of filesystem "/dev/fs1"
    [2015-04-28 03:55:33] Scanning inodes, phase 1 ...
    [2015-04-28 03:55:33] 49907392 inodes have been scanned, 100% of total.
    [2015-04-28 03:55:33] There's no data changes since the restoring snapshot, skipping restore.
    [2015-04-28 03:55:33] Restore completed successfully.
    [2015-04-28 03:55:33] Clean up.

   Run applyUpdates to sync up primary to acting primary:

/usr/lpp/mmfs/bin/mmafmctl fs1 applyUpdates -j newgpfs1022


    [2015-04-28 03:55:36] Getting the list of updates from the acting primary...
    [2015-04-28 03:55:45] Applying the 3 updates...
    [2015-04-28 03:55:47] 3 updates have been applied, 100% of total.
    Flushing dirty data for snapshot newgpfs1022::failback-psnap-rpo-C0A802135530BD89-187-15-04-28-03-55-35...
    Quiescing all file system operations.
    Snapshot newgpfs1022::failback-psnap-rpo-C0A802135530BD89-187-15-04-28-03-55-35 created with id 333.

    Complete failback process on new primary:

/usr/lpp/mmfs/bin/mmafmctl fs1 failbackToPrimary -j newgpfs1022 --stop


    Fileset newgpfs1022 changed.

Convert the acting primary back to secondary and re-establish the relationship. Relink the fileset for this step.

/usr/lpp/mmfs/bin/mmunlinkfileset homefs1 gpfs1022

/usr/lpp/mmfs/bin/mmchfileset homefs1 gpfs1022 -p


     afmmode=secondary,afmPrimaryID=3235816757204815584-C0A802135530BD89-187
    Fileset gpfs1022 changed.

/usr/lpp/mmfs/bin/mmlinkfileset homefs1 gpfs1022 –J /gpfs/homefs1/gpfs1022


    Primary contents:

ls -l /gpfs/fs1/newgpfs1022


    total 1433600
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_gpfspri_4

    Secondary contents:

ls -l /gpfs/homefs1/gpfs1022


    total 921600
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_actingpri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_actingpri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:55 file_actingpri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_1
    -rw-r--r-- 1 root root 104857600 Apr 28 03:53 file_gpfspri_2
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_3
    -rw-r--r-- 1 root root 104857600 Apr 28 03:54 file_gpfspri_4