Role reversal

This use case describes AFM DR role reversal process where secondary site/cluster takes over the role of a new primary site after a failure is observed at the primary site. After the old primary site is back online, the old primary is promoted to secondary site and replication between new primary and secondary site resumes. Roles of both sides are swapped.

In this example, name of the primary fileset is originalPrimary and the name of the secondary fileset is originalSecondary.

On the primary cluster, create a primary fileset.

 mmcrfileset fs1 originalPrimary -p afmTarget=c41bn4:/gpfs/fs1/originalSecondary,afmMode=primary --inode-space=new –afmRPO=12H

A sample output is as follows:

Fileset originalPrimary created with id 117 root inode 24641539. Primary Id (afmPrimaryID) 15220262555331187066-C0A8291B55B52003-117

On the secondary cluster, create a secondary fileset and provide the primary ID in step 1.

 mmcrfileset fs1 originalSecondary -p afmMode=Secondary,afmPrimaryID=15220262555331187066-C0A8291B55B52003-117 --inode-space=new

A sample output is as follows:

Fileset originalSecondary created with id 102 root inode 94371843.

Link the secondary fileset and export it over NFS.

 mmlinkfileset fs1 originalSecondary -J /gpfs/fs1/originalSecondary

A sample output is as follows:

Fileset originalSecondary linked at /gpfs/fs1/originalSecondary

Set up the NFS export of the originalSecondary fileset.

# echo "/gpfs/fs1/originalSecondary *(rw,nohide,insecure,no_subtree_check,sync,no_root_squash,fsid=157322)" >> /etc/exports 
# exportfs –ra 
# showmount -e | grep original

A sample output is as follows:

/gpfs/fs1/originalSecondary *

On the primary cluster, link the primary fileset.

 mmlinkfileset fs1 originalPrimary -J /gpfs/fs1/originalPrimary

A sample output is as follows:

Fileset originalPrimary linked at /gpfs/fs1/originalPrimary
First snapshot name is psnap0-rpo-C0A8291B55B52003-117
Flushing dirty data for snapshot originalPrimary:psnap0-rpo-C0A8291B55B52003-117... Quiescing all file system operations. Snapshot originalPrimary:psnap0-rpo-C0A8291B55B52003-117 created with id 12464.

Check the primary fileset status.

 mmafmctl fs1 getstate -j originalPrimary

A sample output is as follows:

Fileset Name     Fileset Target                         Cache          Gateway     Queue   Queue
                                                        State          Node        Length  numExec
------------    ---------------------------------------  ------        ----------  ------- --------
originalPrimary nfs://c41bn4/gpfs/fs1/originalSecondary PrimInitInProg <GW Node>   1        0

Check the primary fileset status again.

 mmafmctl fs1 getstate -j originalPrimary

A sample output is as follows:

Fileset Name     Fileset Target                         Cache    Gateway     Queue   Queue
                                                        State    Node        Length  numExec
------------    ---------------------------------------  ------  ----------  ------- --------
originalPrimary nfs://c41bn4/gpfs/fs1/originalSecondary Active   <GW Node>   0        1

On the primary cluster, create data in the active primary fileset.

# mkdir /gpfs/fs1/originalPrimary/originalPrimaryData_Iteration1 for i in `seq 10` ; do

A sample output is as follows:

 dd if=/dev/urandom of=/gpfs/fs1/originalPrimary/originalPrimaryData_Iteration1/file${i} bs=1M count=1 2> /dev/null ; done

Check the primary fileset status.

 mmafmctl fs1 getstate -j originalPrimary

A sample output is as follows:

Fileset Name     Fileset Target                          Cache    Gateway     Queue   Queue
                                                         State    Node        Length  numExec
------------    ---------------------------------------  ------  ----------  ------- --------
originalPrimary nfs://c41bn4/gpfs/fs1/originalSecondary  Dirty   <GW Node>   30            2

Check the primary fileset status again.

 mmafmctl fs1 getstate -j originalPrimary

A sample output is as follows:

Fileset Name     Fileset Target                          Cache    Gateway     Queue   Queue
                                                         State    Node        Length  numExec
------------    ---------------------------------------  ------  ----------  ------- --------
originalPrimary nfs://c41bn4/gpfs/fs1/originalSecondary  Dirty   <GW Node>    0            32

You can verify that the data is synchronized correctly by using tools such as md5sum.

On the primary, create an RPO psnap.

 mmpsnap fs1 create -j originalPrimary --rpo

A sample output is as follows:

Flushing dirty data for snapshot originalPrimary:psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38... Quiescing all file system operations. Snapshot originalPrimary:psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 created with id 12466.
 mmpsnap: The peer snapshot psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 is created successfully.

Check the primary fileset status.

 mmafmctl fs1 getstate -j originalPrimary

A sample output is as follows:

Fileset Name     Fileset Target                          Cache    Gateway     Queue   Queue
                                                         State    Node        Length  numExec
------------    ---------------------------------------  ------  ----------  ------- --------
originalPrimary nfs://c41bn4/gpfs/fs1/originalSecondary  Active   <GW Node>   0       63

List snapshots on the primary.

 mmlssnapshot fs1 -j originalPrimary

A sample output is as follows:

Snapshots in file system fs1: 
Directory                                        SnapId  Status  Created                  Fileset
psnap0-rpo-C0A8291B55B52003-117                  12464   Valid   Fri Feb 24 02:01:01 2017 originalPrimary
psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 12466   Valid   Fri Feb 24 02:17:39 2017 originalPrimary

List snapshots on the secondary.

 mmlssnapshot fs1 -j originalSecondary

A sample output is as follows:

Snapshots in file system fs1:
Directory                                        SnapId  Status  Created                  Fileset
psnap0-rpo-C0A8291B55B52003-117                  17710   Valid   Fri Feb 24 02:01:18 2017 originalSecondary
psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 17711   Valid   Fri Feb 24 02:17:40 2017 originalSecondary

Delete all the RPO snapshots on the current primary, except the last one.

 mmpsnap fs1 delete -s psnap0-rpo-C0A8291B55B52003-117 -j originalPrimary

A sample output is as follows:

mmpsnap: The peer snapshot psnap0-rpo-C0A8291B55B52003-117 is deleted successfully.

Wait until the fileset status is Active.

 mmafmctl fs1 getstate -j originalPrimary

A sample output is as follows:

Fileset Name     Fileset Target                          Cache    Gateway     Queue   Queue
                                                         State    Node        Length  numExec
------------    ---------------------------------------  ------  ----------  ------- --------
originalPrimary nfs://c41bn4/gpfs/fs1/originalSecondary  Active  c41bn7       0      64

List snapshots on the primary.

 mmlssnapshot fs1 -j originalPrimary

A sample output is as
follows:

Snapshots in file system fs1: 
Directory                                        SnapId  Status  Created                  Fileset
psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 12466   Valid   Fri Feb 24 02:17:40 2017 originalPrimary

List snapshots on the secondary.

 mmlssnapshot fs1 -j originalSecondary

A sample output is as follows:

Snapshots in file system fs1: 
Directory                                        SnapId  Status  Created                  Fileset
psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 17711   Valid   Fri Feb 24 02:17:40 2017 originalSecondary

On primary, unlink the primary fileset.

 mmunlinkfileset fs1 originalPrimary

A sample output is as follows:

Fileset originalPrimary unlinked

On the secondary, trigger a failover to the secondary by using either –restore/--norestore as per requirement. Also, remove the NFS export that is made of this fileset.

 mmafmctl fs1 failoverToSecondary -j originalSecondary --norestore

A sample output is as follows:

mmafmctl: failoverToSecondary not restoring from any previous RPO snapshot Primary Id (afmPrimaryID) 5492471941218300346-C0A873A4531DF78D-102 Fileset originalSecondary changed. Promoted fileset originalSecondary to Primary

List the fileset on the secondary.

 mmlsfileset fs1 originalSecondary --afm -L

A sample output is as follows:

Filesets in file system 'fs1': 

Attributes for fileset originalSecondary:
========================================== 
Status                                                  Linked 
Path                                                    /gpfs/fs1/originalSecondary
Id                                                      102
Root inode                                              94371843
Parent Id                                               0
Created                                                 Fri Feb 24 01:56:09 2017
Comment Inode space                                     45
Maximum number of inodes                                100352
Allocated inodes                                        100352
Permission change flag chmodAndSetacl afm-associated    Yes 
Target -- Mode                                          primary 
Async Delay                                             15 (default)
Recovery Point Objective                                720 minutes
Last pSnapId                                            0
Number of Gateway Flush Threads                         4 
Primary Id                                              5492471941218300346
                                                        -C0A873A4531DF78D-102

Remove the NFS export to prevent any changes from the old primary. Ensure the NFS export is not available by issuing the following command:
```
# showmount -e | grep original
```

On primary, disable AFM on the unlinked primary fileset, link the fileset back and convert the fileset to secondary with primary ID from the promoted secondary fileset (acting primary). Ensure you export this fileset over NFS.

 mmchfileset fs1 originalPrimary -p afmTarget=disable

A sample output is as follows:

Fileset originalPrimary changed.

List the primary filesets.

 mmlsfileset fs1 originalPrimary --afm -L

A sample output is as follows:

Filesets in file system 'fs1':

Attributes for fileset originalPrimary: 
======================================== 
Status                                                 Linked 
Path                                                   /gpfs/fs1/originalSecondary 
Id                                                     117 
Root inode                                             24641539
 Parent Id                                             -- 
Created                                                Fri Feb 24 01:53:37 2017 
Comment 
Inode space                                            47 
Maximum number of inodes                               100032 
Allocated inodes                                       100032 
Permission change flag chmodAndSetacl afm-associated   No

Link the primary filesets.

 mmlinkfileset fs1 originalPrimary -J /gpfs/fs1/originalPrimary

A sample output is as follows:

Fileset originalPrimary linked at /gpfs/fs1/originalPrimary

Check the NFS export of the fileset.
```
# showmount -e | grep original
```

Convert the primary fileset to the secondary fileset.

 mmafmctl fs1 convertToSecondary -j originalPrimary –primaryId 5492471941218300346-C0A873A4531DF78D-102

A sample output is as follows:

Fileset originalPrimary changed.

List the primary filesets.

 mmlsfileset fs1 originalPrimary --afm -L

A sample output is as follows:

Filesets in file system 'fs1': 

Attributes for fileset originalPrimary:
========================================
Status                                                  Linked 
Path                                                    /gpfs/fs1/originalPrimary 
Id                                                      117 
Root inode                                              24641539 
Parent Id                                               0 
Created                                                 Fri Feb 24 01:53:37 2017
Comment 
Inode space                                             47 
Maximum number of inodes                                100032 
Allocated inodes                                        100032 
Permission change flag chmodAndSetacl afm-associated    Yes
Associated Primary ID                                   5492471941218300346-C0A873A4531DF78D-102
Mode                                                    secondary
Last pSnapId                                            0

On secondary (old primary), enable export to that primary can use this as target.

# echo "/gpfs/fs1/originalPrimary *(rw,nohide,insecure,no_subtree_check,sync,no_root_squash,fsid=12675)" >> /etc/exports # exportfs -ra # showmount -e | grep original /gpfs/fs1/originalPrimary

Now, on primary (acting primary) perform changeSecondary operation with --new-target pointing to the above created secondary fileset with --inband option.

 mmafmctl fs1 changeSecondary -j originalSecondary --new-target <primary>:/gpfs/fs1/originalPrimary --inband

A sample output is as follows:

mmafmctl: Caching operations are currently disabled for fileset originalSecondary This is a promoted primary without a target Changing target to new secondary... Fileset originalSecondary changed. Data will be moved to secondary via AFM psnap will be taken at secondary after data movement completes

Now, wait until the fileset status is ‘Active’.

 mmafmctl fs1 getstate -j originalSecondary

A sample output is as follows:

Fileset Name     Fileset Target                          Cache    Gateway     Queue   Queue
                                                         State    Node        Length  numExec
------------    ---------------------------------------  ------  ----------   ------ --------
originalSecondary nfs://c41bn7/gpfs/fs1/originalPrimary  Active   <GW Node>    0     25

On primary, list the filesets.

 mmlsfileset fs1 originalSecondary --afm -L

A sample output is as follows:

Filesets in file system 'fs1': 

Attributes for fileset originalSecondary: 
========================================== 
Status                                                    Linked 
Path                                                      /gpfs/fs1/originalSecondary 
Id                                                        102 
Root inode                                                94371843 
Parent Id                                                 0 
Created                                                   Fri Feb 24 01:56:09 2017 
Comment 
Inode space                                               45 
Maximum number of inodes                                  100352 
Allocated inodes                                          100352 
Permission change flag chmodAndSetacl afm-associated      Yes 
Target                                                    nfs://c41bn7/gpfs/fs1/originalPrimary 
Mode                                                      primary 
Async Delay                                               15 
Recovery Point Objective                                  disable 
Last pSnapId                                              17712 
Number of Gateway Flush Threads                           4 
Primary Id                                                5492471941218300346
                                                          -C0A873A4531DF78D-102

On the primary, list the snapshots.

 mmlssnapshot fs1 -j originalSecondary

A sample output is as follows:

Snapshots in file system fs1: 
Directory                                        SnapId  Status  Created                  Fileset
psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 17711   Valid   Fri Feb 24 02:17:40 2017 originalSecondary
psnap0-rpo-C0A873A4531DF78D-102                  17712   Valid   Fri Feb 24 02:34:08 2017 originalSecondary

On the current secondary, list the filesets.

 mmlsfileset fs1 originalPrimary --afm -L

A sample output is as follows:

Filesets in file system 'fs1': 

Attributes for fileset originalPrimary:
======================================== 
Status                                                  Linked 
Path                                                    /gpfs/fs1/originalPrimary 
Id                                                      117 
Root inode                                              24641539
Parent Id                                               0
Created                                                 Fri Feb 24 01:53:37 2017
Comment Inode space                                     47
Maximum number of inodes                                100032
Allocated inodes                                        100032
Permission change flag chmodAndSetacl afm-associated    Yes 
Associated Primary ID                                   5492471941218300346-C0A873A4531DF78D-102
Mode                                                    secondary 
Last pSnapId                                            0

On the current secondary, list the snapshots.

 mmlssnapshot fs1 -j originalPrimary

A sample output is as follows:

Snapshots in file system fs1: 
Directory                                        SnapId  Status  Created                  Fileset
psnap-rpo-C0A8291B55B52003-117-17-02-24-02-17-38 12466   Valid   Fri Feb 24 02:17:39 2017 originalPrimary
psnap0-rpo-C0A873A4531DF78D-102                  12467   Valid   Fri Feb 24 02:34:46 2017 originalPrimary