Repairing data block replica mismatches with the file level replica selection rule

Follow this procedure if you want to select the reference data block replica among mismatched replicas on a per file basis.

This method allows for more granular control over which data block replicas are read for a file. Any user who has permission to write on the file can use this method. Before you start, make sure that you determine the list of the mismatched replicas of the file with one of the following commands:

mmrestripefs <fs> --check-conflicting-replicas #requires root privileges
mmrestripefile -c --read-only <Filename> #requires read permission on the file
You can figure out the correct data block replicas among the mismatched replicas of the file by setting a file-specific replica selection rule. This rule is in the form of an extended attribute that is called readReplicaRule, which is under the gpfs namespace. This rule causes a subsequent read of the file to return the data block replicas as specified by the rule. You can then validate the file data by processing it through an associated application.
Note: Setting this extended attribute invalidates existing caches of the file so that subsequent reads of the file fetch the data from disk.
  1. Set the gpfs.readReplicaRule extended attribute with one of the following methods: 
    mmchattr --set-attr gpfs.readReplicaRule=<"RuleString"> <FilePath>
    setfattr -n gpfs.readReplicaRule -v <"RuleString"> <FilePath>
  2. You can also set the extended attribute on the files in a directory tree by using a policy rule such as the following example: 
    RULE EXTERNAL LIST 'files' EXEC ''
RULE LIST 'files' DIRECTORIES_PLUS ACTION(SetXattr('ATTR', 'ATTRVAL'))

    Which can then be applied in a manner such as the following example:

    mmapplypolicy <dir> -P <policyRuleFile> -I defer -f /tmp -L 0 -M ATTR="gpfs.readReplicaRule" -M ATTRVAL="<RuleString>"
  3. You can also set the extended attributed by using an inode number instead of the file name: 
    mmchattr --set-attr gpfs.readReplicaRule=<"RuleString"> --inode-number <SnapPath/InodeNumber>
    Note: This action requires root privilege.
    Where SnapPath is the path to the snapshot root directory, which contains the InodeNumber with replica mismatches. If the replica mismatch is for a file in the active file system, then SnapPath would be the path of the root directory of the active file system. For example:
    mmrestripefile -c --inode-number /gpfs/fs1/.snapshots/snap2/11138
mmrestripefile -c --inode-number /gpfs/fs1/11138

    The gpfs.readReplicaRule extended attribute can be set on any valid file (not a directory or soft link) including clone parents, snapshot user files, and files with immutable or append-only flags. In most cases, to set this extended attribute, you need permission to write on the file. However, if you own the file, you can set the attribute even if you have only the permission to read the file. This attribute can be set even on a read-only mounted file system.

    This attribute is specific to a file, so it does not get copied during DMAPI backup, AFM, clone creation, or snapshot copy-on-write operations. Similarly, this attribute cannot be restored from a snapshot.

If you do not have enough space to store the gpfs.readReplicaRule extended attribute, then you can temporarily delete one or more of the existing user-defined extended attributes. After you repair the replica mismatches in the file, you can delete the gpfs.readReplicaRule extended attribute and restore the earlier user-defined attributes.

To save and restore all of the extended attributes for a file, run the following commands:

getfattr --absolute-names --dump -m "-" <FilePath> > /tmp/attr.save
setfattr –restore=/tmp/attr.save
Note: It is possible to filter out all of the replicas of a data block by using the gpfs.readReplicaRule extended attribute. In such a case, the block read fails with I/O error.

Files with the gpfs.readReplicaRule extended attribute might experience a small impact on read performance because of parsing the rule string for every data block that is read from the disk. Thus, it is advised to delete the gpfs.readReplicaRule extended attribute after you repair the data block replica mismatches.