Pre-migrating files to the cloud storage tier

Using Transparent cloud tiering, you can migrate files in both co-resident and non-resident states depending on the type of data (warm/cold).

Normally, when a file is migrated to the cloud storage tier, the status of the file becomes non-resident. This means that, you need to completely recall the file from the cloud to be able to perform any read or write operation. This might be an issue when the data is warm. The calling application must recall the file every time it needs to perform any operation on the file, and this can be resource-intensive. The solution to this issue is, migrating files in co-resident state or pre-migration.

In this type of migration, irrespective of the file size, when the files are warm, they are archived to the cloud storage in the co-resident state. That allows applications to have continued access to the files without issuing any recall commands, but at the same time, ensures that data is at least available on the cloud if there is any type of disaster. As data gets colder, the files are migrated in the non-resident state. Since files are available both in the file system and on the cloud, the storage utilization is more here.

Small files whose data resides entirely in the inodes are migrated in the co-resident state even if they are cold.

Note: When files are pushed through NFS that need to be moved immediately to the cloud tier, be aware that CES NFS keeps NFSv3 files open and keeps them open indefinitely for performance reasons. Any files that are cached in this manner will not be migrated to by Transparent cloud tiering to the cloud tier. NVSv4 client caching is more measured and less likely to prevent files from being migrated to the cloud tier and is recommended for this sort of usage.

You can migrate files in the co-resident state by using a policy as well as the CLI.

You can use the following command to migrate files in the co-resident state, where the --co-resident-state keyword is mandatory:

mmcloudgateway files migrate --co-resident-state file1

To verify that the file is migrated in the co-resident state, issue the following command:

mmcloudgateway files list file1

The system displays an output similar to this:


File name    : /gpfs0/file1
On-line size : 46
Used blocks  : 0
Data Version : 1
Meta Version : 1
State        : Co-resident
Base Name    : 57FA294111831B2B.10D9F57158C1628B.0063C1580F522F09.0000000000000000.5713748E.0000000000009E2B

For transparent migration in the co-resident state, the following policy has to be applied to the files by using the mmchpolicy command. A sample policy is available here: /opt/ibm/MCStore/samples/coresidentMigrate.template:


/*******************************************************************************
 * Licensed Materials - Property of IBM
 *  
 *  OCO Source Materials
 *  
 *  (C) Copyright IBM Corp. 2017 All Rights Reserved
 *  
 *  The source code for this program is not published or other-
 *  wise divested of its trade secrets, irrespective of what has
 *  been deposited with the U.S. Copyright Office.
 *******************************************************************************/
 define(
        exclude_list,
        (
         FALSE
         OR PATH_NAME LIKE '%/.mcstore/%'
         OR PATH_NAME LIKE '%/.mcstore.bak/%'
        )
)

/* Define premigrate pool, where files are migrated in co-resident state. This represent files moved 
to cloud but also available locally on Scale file system.
* It is to be used for warmer data, as that data needs to be available locally on Scale file system 
too, to avoid cloud round trips.
*/
RULE EXTERNAL POOL 'premigrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '--co-resident-state -F'

/* Define migrate pool, where files are migrated in non-resident state. This represent files are moved
to cloud and are not available locally.
 * It is to be used for colder data depending on file size. Larger colder files are made non-resident,
where as smaller files (less than 4K) are kept co-resident.
 */
RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F'

/* This rule defines movement of warm data. Each file (irrespective of it's size) is moved to cloud 
in a co-resident state.
 * It means, file is available on the cloud and, access to it is possible from the hot-standby site 
if needed.
 * Here the sample time interval to indicate warm data is, data that is not accessed between 10 
to 30 days.
 * We don't want to pick up HOT data that is being accessed in last 10 days.
 * Another advantage of this co-resident migration is when data eventually gets colder, since it
is already migrated to cloud, only file truncation happens later.
 */
RULE 'MoveWarmData' MIGRATE FROM POOL 'system'
	THRESHOLD(0,0)
	TO POOL 'premigrate'
		WHERE NOT(exclude_list) AND
			(CURRENT_TIMESTAMP - ACCESS_TIME > INTERVAL '10' DAYS) AND
			(CURRENT_TIMESTAMP - ACCESS_TIME < INTERVAL '30' DAYS)


/* This rule defines movement of large files that are cold. Here, files that are above 4KB in size
are made non-resident to save
 * space on Scale file system. For files that are smaller than 4KB are anyway stored in inode 
block itself.
 */
RULE 'MoveLargeColdData' MIGRATE FROM POOL 'system'
	THRESHOLD(0,0)
	TO POOL 'migrate'
		WHERE(KB_ALLOCATED > 4) AND NOT(exclude_list) AND
			(CURRENT_TIMESTAMP - ACCESS_TIME > INTERVAL '30' DAYS)

/* This rule defines movement of smaller files that are cold. Here, files that are less than 
4KB in size are made co-resident, as
* there is no saving in moving these files, as data resides within the inode block, and not 
on disk. It avoids un-necessary recall cycles.
*/
RULE 'MoveSmallColdData' MIGRATE FROM POOL 'system'
	THRESHOLD(0,0)
	TO POOL 'premigrate'
		WHERE(KB_ALLOCATED < 4) AND NOT(exclude_list) AND
			(CURRENT_TIMESTAMP - ACCESS_TIME > INTERVAL '30' DAYS)

/* Define default placement rule */
RULE 'Placement' SET POOL 'system'