Backup considerations for Transparent cloud tiering

You must adhere to some guidelines for backing up files that need to be migrated to a cloud storage tier.

In many cases, Transparent cloud tiering (cooler data) might be run along with file system backups (for hot and warm data) for data protection. Transparent cloud tiering should not be used as a replacement for a valid backup strategy. Files that are migrated with Transparent cloud tiering might be lost if accidentally deleted or if there is a complete site failure. File system backups can protect from these types of events.

When you work with Transparent cloud tiering file system backups, it is important that a file is backed up before it is sent to a cloud storage. A file that is not backed up is recalled from the cloud storage tier when a backup program is run. This can cause unnecessary recall traffic, and also cause the backup to take longer to run. Recalling data also involves cost if you are using a public cloud provider.

To ensure that files are not migrated before they are backed up, it is best to define a migration policy that explicitly excludes files that are recently modified. For example, if nightly backups are run, at minimum, a migration policy should exclude files that are modified within the past day, so they can be backed up before they are migrated. In most cases, it is best to wait longer to avoid recalls if a backup window is missed for some reason. For example, if you run a nightly backup, it is advisable to exclude files that are modified in the past 3 days. If there is an issue with running backups for a long period, it might be necessary to disable a migration policy until the problem with backups is resolved. This avoids many recalls during the next successful backup.

The following is a sample migration policy and it should be tuned as per the backup frequency that is configured for an IBM Storage Scale file system.

Note: The amount of time to wait to migrate a file should be greater than the frequency of file system backups.

For information on backing up configuration data (which is critical for disaster recovery), see Backing up the Cloud services configuration.

For information on backing up the cloud database, see Backing up the Cloud services database to the cloud.

You can exclude recently modified files with a policy statement such as the following:


WHERE (DAYS(CURRENT_TIMESTAMP) - DAYS(MODIFICATION_TIME)) > 5

This statement can also be added for manual application of the policy for a directory.


/* Define a cloud pool, using the ‘mmcloudgateway’ command to perform migrations to the pool */
   RULE EXTERNAL POOL ‘cloudpool’ EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F'


/* Define a migrate rule, in this example we will migrate from the system pool */
   RULE ‘MigrateToCloud’ MIGRATE FROM POOL ‘system’    
/* Define the threshold to use, in this example we will migrate when the pool is 85% full, 
   and attempt to migrate files until the pool is 75% full */
   THRESHOLD (85, 75)    
/* Next, define a weight for the file to determine which files to migrate first */ 
/* Here we will use the last access time for the file by subtracting it from the current time,
   giving a higher weight to older files */
   WEIGHT(CURRENT_TIMESTAMP - ACCESS_TIME)
/* Define that we want to move to the cloud pool */   
   TO POOL (‘cloudpool’)
/* We don’t want to migrate files that are smaller than 8k in size, because there are little to
   no space savings due to file overhead */
   WHERE (KB_ALLOCATED > 8)    
/* Do not migrate Cloud services internal files, which reside in the .mcstore directory */
   AND NOT(PATH_NAME LIKE ‘%/.mcstore/%’  
/* Ignore files that have been modified in the past 5 days. This needs to be customized based on
   the backup frequency. */ 
   AND (DAYS(CURRENT_TIMESTAMP) - DAYS(MODIFICATION_TIME)) > 5