Replica mismatches

IBM Storage Scale includes logic that ensures data and metadata replicas always have identical content.

A replica mismatch is a condition in which two or more replicas of a data or metadata block differ from each other. Replica mismatches might happen for the following reasons:

Corrupted log files (replica writes are protected by IBM Storage Scale journals) found by IBM Storage Scale.
Hardware issues such as a missed write or a redirected write that led to stale or corrupted disk blocks.
Software bugs in administrative options for commands such as mmchdisk start and mmrestripefs that fail to sync the replicas.

When a replicated block has mismatched replicas, the wrong replica block can be read and cause further data integrity issues or application errors.

If the mismatch is in a metadata block and is found by IBM Storage Scale to be corrupted, it is flagged in the system log as an FSSTRUCT error. The next replica block is then read. While this action does not disrupt file system operations, it leaves the metadata block with an insufficient number of good replicas. A failure of the disk that contains the good replica can lead to data or metadata loss. Alternatively, if the replica block that is read contains valid but stale metadata, then it can lead to further corruption of the data and metadata. For instance, if the block belongs to the block allocation map system file, then reading a stale replica block of this file means that IBM Storage Scale sees a stale block allocation state. This issue might allow IBM Storage Scale to double allocate blocks, which leads to corruption due to block overwrite. You can identify such a corruption by looking for the FSErrDeallocBlock FSSTRUCT error in the system log, which is logged at the time of the block's deallocation.

If the mismatch is in a data block, then IBM Storage Scale cannot determine whether the replica that is read is corrupted since it does not have enough context to validate user data. Hence, applications might receive corrupted data even when a good replica of the data block is available in the file system.

For these reasons, it is important to be able to repair replica mismatches as soon as they are detected. You can detect replica mismatches with any of the following methods:

With an online replica compare:

mmrestripefs <fs> --check-conflicting-replicas

With a read-only offline fsck:
```
mmfsck <fs> -nv [-c|-m]
```
By comparing the replicas on a per file basis:
```
mmrestripefile -c --read-only <Filename>
```
Note: This option detects data and directory block replica mismatches only.

Any of these methods can be used when file reads return stale or invalid data, or if FSErrDeallocBlock FSSTRUCT errors are in the system logs.

If replica mismatches are detected, then the next step is to make the replicas consistent. The replicas can be made consistent by choosing a reference replica block to copy over the other replicas of the block. If they are metadata block replica mismatches, the reference replica block is chosen by IBM Storage Scale. However, if they are data block replica mismatches, the reference replica should be chosen by the file owner.