Reconciling files between IBM Spectrum Scale file system and cloud storage tier

This topic describes how to reconcile files migrated between IBM Spectrum Scale™ file systems and the cloud tier.

When you remove files from the file system, no policy is currently available to automatically remove the cloud objects that are created, which Start of change results in deleted files or older versions of files that have been migrated to the cloud to accumulate asEnd of change objects on the cloud.

To clean that up, it is recommended to periodically issue the following command to keep the cloud in sync with the file systemStart of change so as not to accumulate an excessive backlog of deletes to be processedEnd of change.

Start of change
mmcloudgateway files reconcile Device {--days-retained DaysRetained}
End of change
where,
  • Device is the device name of the file system that is associated with the node class
  • Start of changeDaysRetained is the number of days for which the deleted files are to be retained on the cloud.
    Note: This option provides a control on which files are deleted during the reconcile operation. Only those files that have been deleted for the specified duration are removed from the database. The rest of the files remain in the database. By doing so, files that have been accidentally deleted for the configured duration can be recovered by using the mmcloudgateway files restore option.
    End of change
If a migrated file is removed from the file system, reconcile removes the corresponding cloud objects and references that are contained in the cloud directory. Additionally, if multiple versions of a file are stored on the cloud, reconcile removes all older cloud versions (keeping the most recent). For example, if a file is migrated, then updated, and migrated again. In this case, two versions of the file are stored on the cloud. Reconcile removes the older version from the cloud. Reconcile also deletes cloud objects that are no longer referenced.
Note: Reconcile removes entries from the cloud directory that references deleted file system objects. Therefore, it is recommended that any files that need to be restored are restored before you run a reconcile. It is also recommended to run the reconciliation operation as a background activity during low load on the IBM Spectrum Scale cluster.