mmafmcosctl command

Controls downloads and uploads between a cloud object storage and an AFM to cloud object storage fileset.

Synopsis

mmafmcosctl Device FilesetName Path download {--object-list ObjectList | --all} 
           [{--metadata | --data} --no-sub-dir --prefix NamePrefix --uid UID --gid GID --perm Permission]
           [--outband[-N Node]] [--threadsNum][--enable-failed-object-list]

mmafmcosctl Device FilesetName Path upload {--object-list ObjectList | --all} 
            --evict-after-upload

mmafmcosctl Device FilesetName Path evict {--object-list ObjectList | --all} [--metadata][--validate]

mmafmcosctl Device FilesetName Path delete {--from-cache  | --from-target}

mmafmcosctl Device FilesetName {--upload-stats | --download-stats}[-Y]

mmafmcosctl Device FilesetName Path checkTCT

mmafmcosctl Device FilesetName Path reconcile [ --policy path | --add-policy path | --remove-policy | 
          --list-policy | --evict-after-upload]

Availability

Available on all IBM Storage Scale editions.

Description

This command is used to control upload, download, and eviction on an AFM to cloud object storage fileset. For more information, see Filesets.

Parameters

Device

Specifies a device name of a file system.

FilesetName

Specifies a name of an existing AFM to cloud object storage fileset on which the operation is to be performed.

Path

Specifies a path under an AFM to cloud object storage fileset on which the operation needs to be performed.

download

Specifies the download operation that is performed on an AFM to cloud object storage fileset. This operation downloads the objects from a cloud object storage and presents them as files to the AFM to cloud object storage fileset.

upload

Specifies the upload operation that is performed on an AFM to cloud object storage fileset.

On a local-update (LU)-mode fileset that is enabled with the AFM to cloud object storage, you can upload data that was created or modified on the fileset locally (on the cache). You can upload this data to the cloud object storage server as an object. This is supported for the LU mode.

evict

Specifies the eviction operation of the file data (blocks) or file metadata (inode) on an AFM to cloud object storage fileset.

This operation removes file data and/or file metadata (inodes) only from the AFM to cloud object storage. This operation removes only data on AFM to cloud object storage. Objects on AFM to cloud object storage are not removed. This is a local operation.

This operation helps to save space (storage blocks and inodes) so that less priority data will be evicted from the local cache and makes space for priority data as per the quota. You can either evict all files and/or metadata by using the --all option or evict selected data or metadata by using the --object-list option. The objectlist file is a line separated file that contains a full path of files on the fileset.

If require, you can download objects to the AFM to cloud object storage fileset again by using the mmafmcosctl download command.

Eviction will be triggered on the local cache and the objects on the cloud object storage will still have the data or metadata. This is a local operation.

delete

Specifies the delete operation that is performed on an AFM to cloud object storage fileset.

The --from-cache option reclaims or cleans up the inodes of the files that are already deleted locally from the fileset. After the inodes of the files are deleted, –from-cache option does not queue the deletes to cloud object storage .

When the auto remove option is configured by using mmchconfig afmMUAutoRemove command, this feature overrides the –from-cache or –from-target options.

The --from-target option reclaims or cleans up the inodes of the files that are already deleted from the MU mode fileset as well as queue the deleted to cloud object storage to delete the same files from it.

--object-list

Specifies a line separated list of files that you want to download, upload, or evict. Use the option perform operation on selected files. Otherwise, you can use the --all option. This list file of objects might contain relative paths that support upload and download operations.

When you use the evict operation, a full file path name must be specified. You can either specify the --object-list or --all option.

--upload-stats

Shows upload progress for the previously run or currently running upload task executed by using mmafmcosctl upload command.

--download-stats

Shows download progress for the previously run or currently running download task executed by using mmafmcosctl download command. The --download-stats option will not show output for download --metadata and stat will display output for only queued objects.

-Y

Displays the command output in a parseable format with a colon (:) as a field delimiter. Each column is described by a header. The '-Y' option is applicable with the '--upload-stats' and '--download-stats' parameters.

--all

Specified with upload, download and evict parameters. This option performs the upload, download, or evict operation on all the data inside the given fileset. AFM generates internally a list of all the files and performs the operation on the list.

--metadata

Specified with download and evict operations. The option enables the AFM to cloud object storage to download or evict metadata.

When this option is specified with the download, only the objects metadata information is downloaded from the cloud object storage. This helps to populate all the directory tree structure on the cache from the bucket.

You can run the download operation along with the --data option to populate data blocks. This is the default option for downloading objects.

When this option is specified with the evict operation, both the data blocks and file inodes are evicted only from the AFM fileset. However, data blocks and file inode information will still be available in the bucket on cloud object storage server. This makes the space on the cache fileset for required data. This parameter is optional for the evict operation. If this option is not specified for eviction, default is to evict data blocks.

--validate

Specified with the evict operation. When this option is specified with the evict operation for a file, AFM verifies whether the same file is available in a cloud object storage bucket. If the file is available in the bucket, then AFM evicts the data blocks of the file on the AFM cache site. If the file is not available in the cloud object storage bucket, then the evict operation fails and the data block of the file is not evicted. End of change

--data

Specified with the download operation. This option enables the AFM to cloud object storage to download data of the given list. The --data option downloads metadata implicitly.

--no-sub-dir

Specifies the AFM to cloud object storage to skip downloading the sub directories from a cloud object storage. You can skip this option to download the sub directories from a cloud object storage.

--prefix

Specifies a prefix for an object on a specified directory on an AFM to cloud object storage fileset.

--uid

Specifies a user ID to be set on an AFM to cloud object storage fileset. If not specified, the owner is set as default, for example, root.

--gid

Specifies a group ID to be set on an AFM to cloud object storage fileset. If not specified, the group is set as default, for example, root.

--perm

Specifies the access permission to be set on an AFM to cloud object storage fileset. This is in the octal format, for example, 0770. If not specified, the permission is set to default.

--outband

Specifies the --outband option to download the objects or files from the cloud object storage target by using the outband method. In this method, any node that can connect to the cloud object storage can be used to download the files. In outband download method, multiple threads can be used to download files. Outband download method does not use AFM gateway interface as well as VFS operations that serializes queues for the performance enhancement.

When the --outband option is specified, fileset can remain inactive and there is no gateway queue built.

-N Node

Specifies the node that is used to download the objects or files by using outband method when -N is not provided by the default gateway node for the fileset that is used. To download from the cloud object storage the node that is specified need access to the cloud object storage buckets.

--threads Num

Specifies the number of threads that can be used to download the objects by using the outband method. Threads from value 1 to 512 are supported.

--enable-failed-object-list

When the --enable-failed-object-list option is provided, the outband download command creates a list of files that are failed to download from the cloud object storage. The file list is generated on the node specified by -N node parameter. When -N node is not provided, the list is generated on the gateway node.

reconcile

Specifies the reconcile operation by using user defined policy on an AFM to cloud object storage fileset. This reconcile operation uses the policy provided by the system administrator and uploads the files or objects according to the criteria defined in the policy. The reconcile command runs user defined policy if it is installed already by using –add-policy option.

The reconcile command also performs clean-up of files when the command is run along with the policy-based uploads. Files that are deleted from manual update mode fileset are queued for deletes on the Cloud Object Storage and their inodes are reclaimed.

checkTCT

Processes the converted TCT-enabled file system or fileset to the AFM manual update (MU) mode after the mmafmcosconfig conversion is complete. End of change

After the TCT-enabled fileset or file system is converted to the AFM MU mode by using the mmafmcosconfig --convert command, the checkTCT option runs a query to collect the non-resident, co-resident files, which were synced to a cloud object storage bucket and generates a list-file. The generated list file is used to run lookup to queue them to the AFM gateway node. After the data synchronization of AFM MU mode fileset with the cloud object storage bucket is complete, AFM removes the dmapi.MCEA attribute from the collected list-file entries. The AFM MU mode fileset manages synchronization after the TCT relationship of file is disabled. End of change

After files are queued on the AFM gateway node, AFM removes the TCT object structure information from the cloud object storage bucket. TCT object structure information is replaced with the AFM style object structure information to the storage data in the bucket. End of change

An example of the checkTCT option is as follows:

Generate a list of files by using the checkTCT option.
```
 # mmafmcosctl fs1 tctFileset /gpfs/fs1/tct4 checkTCT
```
A list of files tiered by using the TCT for the tct4 - /gpfs/fs1/tct4/.mcstore/.mceaEnabledFiles.list.mceaFiles.3770335 fileset.
Run lookup on all the files from the generated list-file.
```
 cat /gpfs/fs1/tct4/.mcstore/.mceaEnabledFiles.list.mceaFiles.3770335 | xargs ls
```
This command queues a lookup request on the AFM gateway node. When all the request of the specified fileset are complete, AFM removes dmapi.MCEA extended attribute from the list-files.

Check whether the dmapi.MCEA is removed the list-files.

# mmlsattr -d -L /gpfs/fs1/tct4/file1.txt|grep dmapi.MCEA

After the AFM MU mode fileset is in sync, operations such as upload, download, delete, evict, or reconcile can be performed on the fileset. End of change

--policy path

Run reconcile command with the user defined policy. System administrator must provide policy with right syntax and semantics.

--add-policy path

Install user defined policy in AFM. System administrator must provide policy with right syntax and semantics. This policy is stored internally and whenever reconcile command is run without any parameters, the command gets executed.

--remove-policy

Removes the policy installer by using –add-policy option.

--list-policy

Shows the policy which is added previously by using --add-policy option.

--evict-after-upload

Specifies that files uploaded to the respective cloud object storage targets will be evicted immediately after the upload operation from the AFM to cloud object storage fileset, thus assisting space utilization of fileset. When this parameter is provided with the reconcile command all the files curated by provided policy or default policy will be uploaded and immediately evicted in the AFM to cloud object storage fileset. The renamed file will not be evicted immediately until data is modified.

Exit status

0: Successful completion.
nonzero: A failure occurred.

Security

The mmafmcosctl command execution does not require a root user. This command enables non-root users to download, upload, or evict data from an AFM to cloud object storage fileset.

The node on which the command is issued must be able to run remote shell commands on any other node in the cluster. The node must run these remote shell commands without a password and must not produce any extraneous messages. For more information, see Requirements for administering a GPFS file system.

Examples

To download data of selected objects from a bucket to the cache fileset by using the object-list file, issue the following command:
```
# mmafmcosctl fs1 SW1 /gpfs/fs1/SW1 download --object-list /home/user01/listfile --data
```

To download all objects from a bucket to the cache, issue the following command:

# mmafmcosctl fs1 SW1 /gpfs/fs1/SW1 download --all --metadata --uid user01 --gid gid01

To evict files or objects data, issue the following command:

# mmafmcosctl fs1 afmtocos1 /gpfs/fs1/afmtocos1/ evict --object-list /root/evictfile

To evict files or objects metadata, issue the following command:

# mmafmcosctl fs1 afmtocos1 /gpfs/fs1/afmtocos1/ evict --object-list /root/evictfile --metadata

To download files by using the --prefix parameter, issue the following command:

# mmafmcosctl fs1 iw1 /gpfs/fs1/iw1/ download --object-list /root/downloadfiles --data --prefix thread_1

A sample output is as follows:

     Queued    (Total)      Failed                TotalData
                                          (approx in Bytes)
          0         (0)          0                        0
         20         (0)          0                   204800
Object Downloads successfully queued at the gateway.

A prefix is a directory inside the /gpfs/fs1/iw1 fileset, and the object-list contains a path for all files under the prefix.

To queue the delete operation on the files manually from the target, issue the following command:
```
# mmafmcosctl fs1 mufilesetdemo /gpfs/fs1/mufilesetdemo/ delete --from-target
```

To upload data from MU to COS, issue the following command:

# mmafmcosctl fs1 demo1bucket /gpfs/fs1/demo1bucket/ upload --object-list/root/uploadobjectlist

A sample out is as follows:


Queued    (Total)      Failed          TotalData
                                     (approx in Bytes)
 3         (3)          0               31457280

Object Upload successfully queued at the gateway.

Location

/usr/lpp/mmfs/bin