Pre-populating metadata by using the out of band prefetch
Sometimes, prefetching of data and metadata from a home target might take more time. During prefetch, AFM queues a lookup operation to a gateway node for each file. This operation might take several minutes or hours based on the amount of data and network latency. If metadata on the home is modified, AFM must re-populate new metadata on an AFM cache fileset.
To overcome this delay, AFM can pre-populate the metadata quickly on the cache fileset by using the metadata list-file. Pre-population by using the list-file does not queue any operation on a gateway node and quickly pre-populates the fileset locally.
The out of band population is a two-step process where metadata list-file needs to be generated on the home site by issuing the mmafmctl getOutBandList command. By using this generated metadata list-file, prefetch can be performed on the cache site by issuing the mmafmctl --outband command.
- The out of band metadata pre-population can only be performed on an AFM fileset by using NSD backed targets. That is, multi-cluster backend. For more information about the NSD backend, see Backend protocol - NFS versus NSD .
- Both home and cache site must be running IBM Storage Scale 5.1.9 or higher.
- Pre-population of metadata is only supported for POSIX metadata only, such as UID, GID, file permissions, size, mtime. It does not support other metadata, such as NFS v4 acl, EA/xattr.
- If the home or source is not an IBM Storage Scale fileset but an IBM Storage Scale file system path, the fileset can be specified as 'root' when you are issuing the prefetch command.
- The prefetch can be run for deleting files with the --delete option first, and then run the --outband prefetch.
Run the following steps in a loop as per requirement until all modified data on the home is prefetched to the cache and both sites are synchronized. You can run a separate prefetch for download the data on-demand.
# mmafmctl fs1 getOutbandList-j home1 --path /gpfs/fs1/home1 Run prefetch, ex. --list-file /gpfs/fs1/home1/.mmmigrateCfg/.mmmigrate.changed.files
Run prefetch, ex. --delete --list-file /gpfs/fs1/home1/.mmmigrateCfg/.mmmigrate.deleted.files
- The deleted list-file contains information about the files/directories, which were deleted from the specified fileset path on the home fileset. This file can be used to delete the data from an AFM independent writer (IW), single writer (SW), local updates (LU), and read only (RO) mode fileset on the cache cluster.
- The changed list-file contains information about the files or directories and its metadata. The metadata can be used to pre-populate an AFM independent writer (IW), single writer (SW), local updates (LU), and read only (RO) mode fileset on the cache cluster by using the prefetch --outband option.