Failback of multiple filesets

Although it is possible to run the failback procedure for up to 50 filesets at once, doing so might result in an extended outage for your cluster. In the following example, the three steps that compose the failback procedure (--start, applyUpdates, and --stop) were timed to demonstrate the length of the process for one fileset:

[root@boris.prt001st001 ~]time mmafmctl gpfs1 failbackToPrimary -j fileset1 --start
real 0m11.486s
user 0m0.004s
sys 0m0.000s

[root@boris.prt001st001 ~]time mmafmctl gpfs1 applyUpdates -j fileset1 
real 30m01.490s
user 1m0.004s
sys 1m0.000s


[root@boris.prt001st001 ~]time mmafmctl gpfs1 failbackToPrimary -j fileset1 --stop
real 0m11.486s
user 0m0.004s
sys 0m0.000s

The applyUpdates step took 30 minutes to complete for one fileset. If you assume 30 minutes per fileset, then it could take up to 25 hours to complete the applyUpdates step for 50 filesets:

for I in {1..50}; do mmafmctl gpfs1 failbackToPrimary -j fileset$I --start ; done
for I in {1..50}; do mmafmctl gpfs1 applyUpdates -j fileset$I  ; done
for I in {1..50}; do mmafmctl gpfs1 failbackToPrimary -j fileset$I --stop ; done

At that point, the primary would be 25 hours behind the secondary and must be run again. One way to avoid the primary being many hours behind the secondary is to take the entire cluster offline for an extended period. However, a better option is to complete the failback process for one fileset at a time so that it is only necessary to take a 30-minute outage:

for I in {1..50};
do
mmafmctl gpfs1 failbackToPrimary -j fileset$I --start ;
mmafmctl gpfs1 applyUpdates -j fileset$I   ;
mmafmctl gpfs1 failbackToPrimary -j fileset$I --stop  ;
done

Using this method, the administrator would be able to select certain filesets to be completed first:

for I in 5 25 30;
do
mmafmctl gpfs1 failbackToPrimary -j fileset --start ;
mmafmctl gpfs1 applyUpdates -j fileset   ;
mmafmctl gpfs1 failbackToPrimary -j fileset --stop  ;
done