mmafmcosconfig command

Availability

Available on all IBM Storage Scale editions.

Description

The mmamfcosconfig command constructs a new AFM to cloud object storage fileset with the specified name parameters. A single command is provided to create and link the AFM to cloud object storage fileset. You can specify the parameters such as policies, mode, and permissions to save running extra commands. To modify any parameters after the creation of the AFM to cloud object storage fileset, you need to run a specific command separately such as establishing policy or modifying quota on the fileset.

A report that has all details of the AFM to cloud object storage fileset can be generated.

Before you run this command, ensure that the gpfs.afm.cos package is installed on the gateway nodes.

For more information about filesets, see the filesets section in the IBM Storage Scale: Administration Guide.

This command can be used to convert an IBM Storage Scale file system or IBM Storage Scale fileset to an AFM to cloud object storage manual update (MU) mode enabled file system or fileset for the cloud synchronization.

Parameters

Device

Specifies a device name of a file system to contain a new fileset.

FilesetName

Specifies a name of an AFM to cloud object storage fileset to be created. To convert a file system to an AFM to cloud object storage fileset for the manual update mode replication use the FilesetName as a root.

Note: The following restrictions should be applied on the fileset names:

• The name must be unique within the file system.
• The length of the name must be in the range 1-255.
• The name root is reserved for a fileset of the root directory of the file system.
• The name cannot be the reserved word "new". However, the character string "new" can appear within a fileset name.
• The name cannot begin with a hyphen (-).
• The name cannot contain these characters: / ? $ & * ( ) ` # | [ ] \fR.
• The name cannot contain a white-space character such as blank space or tab.

--endpoint EndpointURL

Specifies an endpoint URL that is the address of a cloud object storage server on which it receives requests from clients. An endpoint can be either HTTP or HTTPS. It can also be a DNS qualified hostname or an IP address. An endpoint also contains a port number on which the cloud object storage server is running.

Protocol

Specifies the protocol to be used which has either http:// or https:// support .

Region

Specifies a region where a bucket is located on a cloud object server. You can specify a region with a server or a map name by separating them with '@'.

ExportMap

When the parallel reads operation is used for an AFM to cloud object storage fileset, an export map is created by using the mmafmconfig command. This export map must be used instead of the server parameter to perform parallel data operations from the mapped gateways. For more information, see the mmafmconfig command.

port

Specifies a port number on a cloud object storage server on which it is listening from the clients. Default port is 80.

--object-fs

Specifies the behavior of an AFM to cloud object storage fileset. The following two modes can be enabled on an AFM to cloud object storage fileset:

ObjectFS

This behavior is used to enable auto-synchronization of metadata from a cloud object storage server to the AFM to cloud object storage enabled fileset. This mode allows AFM to auto-synchronization fileset metadata when a lookup or readdir operation is performed on the fileset or the AFM refresh intervals are triggered.

AFM downloads information of objects from the cloud object storage automatically and presents it as files on the AFM to cloud object storage fileset. The ObjectFS enabled AFM to cloud object storage fileset behaves in a similar way as an AFM mode fileset on-demand behavior.

For the AFM RO, LU, and IW mode enabled AFM to cloud object storage fileset, AFM automatically synchronizes objects from the cloud object storage to the files on the cache. Modification of objects on cloud object storage will be refreshed on the cache.

For the SW and IW mode enabled AFM to cloud object storage fileset, modification on the cache will queue an upload operation to be synchronized as an object to the cloud storage server.

Enable this mode if you want AFM to cloud object storage fileset behave like an AFM mode fileset. This behavior generates traffic between the cloud object storage server and the AFM to cloud object storage fileset.

ObjectOnly

If the --object-fs parameter is not defined, the ObjectOnly mode will be set. This is the default behavior, which is set on the AFM to cloud object storage server.

With the ObjectOnly mode, refresh of metadata from the cloud object server to an AFM to cloud object storage fileset will not be on-demand or frequent. You need to manually download data or metadata from the cloud object storage to the AFM to cloud object storage fileset.

Meanwhile data synchronization from the AFM to cloud object storage (SW, IW mode) to the cloud object storage will work automatically without manual intervention. This behavior is similar to the AFM fileset mode behavior.

Enable this mode on an AFM to cloud object storage fileset to avoid frequent trips and reduce the network contention by selectively performing the operations.

--convert

Converts the existing GPFS independent fileset to an AFM manual update (MU) mode fileset and establishes relation with a cloud object storage bucket. With --convert option, the other options such as --dir, --uid, --cleanup, and --perm are not supported.

This parameter can be used to convert an existing GPFS file system or GPFS independent fileset, on which TCT is enabled, to an AFM to cloud object storage MU-enabled fileset or file system for the cloud object synchronization with a cloud bucket. For the file system conversion, use the root as a fileset name.

After conversion of TCT to an AFM MU mode fileset, verify data and disable TCT.

--cleanup

Specifies whether to clean up data and delete an existing fileset with the same name, and re-creates a new AFM to cloud object storage fileset with given parameters.

The user confirmation is needed before the data cleanup and the fileset deletion. After the confirmation, the fileset with its data is deleted.

Warning: Fileset 'sw2' will be deleted. Do you wish to continue? (yes/no)

Note: Use this option only if you want to delete an existing fileset with the same name and along with data.

--xattr

Specifies user-extended attributes to be synchronized with a cloud object storage. If this option is skipped, the AFM to cloud object storage does not synchronize user-extended attributes with the cloud object storage.

--ssl-cert-verify

Specifies SSL certificate verification. This option is valid only with HTTPS. Default value of this parameter is disabled.

--user-keys

Specifies adding a callback to collect an access key and a secret key from a file.

When this option is enabled, AFM retrieves the access key and the secrete key by calling the /var/mmfs/etc/mmuid2keys file. The returned value must be in the Access_Key:Secret_Key format.

The mmuid2keys file must be available on the gateway node(s) and must have executable permission.

If you want to create a fileset using Security Token Service (STS) then you must add STS key along with Access_key and Secret_Key in the format as shown in the following example of the mmuid2keys file. The returned value must be in the Access_Key:Secret_Key:STS format.

Example 1: mmuid2keys file

An example of the mmuid2keys file is as follows:

#!/usr/lpp/mmfs/bin/mmksh
                 # input:
                 #
                 #- UID          : User Id configured on the system. It could be local
                 #                 user or LDAP/AD configured user.
                 #- bucket       : The name of the bucket for which keys are required.
                 #- endpoint     : The endpoint of the fileset for which keys are required.
                 #
                 # output:
                 #
                 # accesskey:secretkey
                 # In the output, keys="AccessKey:SecretKey:STS", the 'STS' field is optional.
                 cmd=$1
                 uid=$2
                 bucket=$3
                 endpoint=$4
                 # check for bucket, endpoint and return key
                 if [[  $bucket == "bkt1" && $endpoint =~ "ap-southeast-2@192.168.1.1:443" ]]
                 then
                     keys="AccessKey1:SecretKey1:STS" 
                 elif [[  $bucket == "bkt2" && $endpoint =~ "eu-west-2@192.168.1.2:80" ]]
                 then
                     keys="AccessKey2:SecretKey2:STS"
                 else
                     keys="AccessKey:SecretKey:STS"
                 fi
                 echo "$keys"
                 exit 0

Example 2: mmuid2keys file

In the following simplified example of mmuid2keys file, where all the filesets are created by using --user-keys option and mmuid2keys file returns required credentials as shown.

#!/usr/lpp/mmfs/bin/mmksh
echo "Akey:Skey:STS"   
#The'STS'field is optional.

--bucket BucketName

Identifies a unique bucket on a cloud object storage server. AFM will use this bucket as the target for an AFM to cloud object storage fileset, and it will synchronize data between the AFM to cloud object storage fileset and the bucket on the cloud object storage.

Credentials to access this bucket must be already set by using the mmafmcoskeys command for this bucket.

--new-bucket BucketName

Specifies a new bucket to be created on a cloud object storage by the AFM to cloud object storage and used this as the Target.

The name of the bucket is as per the support of the cloud object storage. Check the cloud object storage for bucket name guidelines.

Credentials must be already set by using the mmafmcoskeys command to create this bucket on the cloud object storage.

--dir Path

Specifies a junction path to link an AFM to cloud object storage fileset inside a file system. If not specified, default junction path will be used.

--policy PolicyFile

Specifies a policy to be applied for an AFM to cloud object storage fileset. If not specified, default policy is enabled.

--tmpdir DirectoryPattern

Specifies a directory pattern if a policy is not provided.

--tmpfile FilePattern

Specifies a file pattern to be set if a policy is not provided.

--quota-files NumberOfFiles

Specifies the number of file limit to be set as file quota on an AFM to cloud object storage fileset. If not specified, default file quota is in effect. Units are in numbers, for example, 104857600.

--quota-blocks NumberOfBlocks

Specifies the data block limits to be set as block quota on an AFM to cloud object storage fileset. If not specified, default block quota is in effect. Units are in numbers.

--uid UID

Specifies a user ID to be set on an AFM to cloud object storage fileset. If not specified, default owner will be set, for example, root.

--gid GID

Specifies a group ID to be set on an AFM to cloud object storage fileset. If not specified, default group will be set, for example, root.

--perm Permission

Specifies the access permission to be set on an AFM to cloud object storage fileset. This is in the octal format. Example 0770. If not specified, default permission will be set.

--mode AccessMode

All AFM fileset modes support an AFM to cloud object storage fileset. These modes are independent-writer (IW), manual-updates (MU), single-writer (SW), read-only (RO), and local-updates (LU). Default is the SW mode.

--fast-readdir

Improves the read-dir performance of an AFM to cloud object storage fileset by skipping synchronization of extended attributes and ACLs of an object between a cloud object storage server and a cache fileset.

When this option is set, deleted objects on the cloud object storage server will not be reflected immediately on the cache fileset.

This option cannot be used with the --xattr option.

This option can be set on an AFM to cloud object storage SW, IW, RO, and LU mode fileset.

--chunk-size Size

Specifies the chunk size to control the number of upload parts on a cloud object storage. The unit is defined in number. Default is 16 MB chunk size.

--read-size Size

Specifies the read size to control the number of download data blocks into an AFM to cloud object storage fileset. The unit is defined in number. Default is 16 MB download size.

--fast-readdir2

Specifies an optional parameter version 2 of --fast-readdir for AFM to perform all the operations of --fast-readdir along with the delete operation synchronization from a cloud object storage bucket to the AFM cache fileset.

When this parameter is specified, AFM pulls minimal metadata during readdir of the objects from the cloud object storage to AFM. Along with pulling metadata, AFM synchronizes the deleted objects from the cloud object storage bucket to the AFM cache. This helps overcome the limitation of --fast-readdir object deletion.

ACL and xattrs (extended attributes) of the object still skip the synchronization to the AFM cache to keep this operation more efficient when only minimal or POSIX metadata is required to be pulled to the AFM cache. This option cannot be set with --acls, --xattr, and --fast-readdir.

This option can be set on an AFM to cloud object storage fileset in the SW, IW, RO, and LU modes.

--acls

Enables the cache to synchronize ACL to the cloud object storage server. If not specified, AFM will not synchronize ACLs to cloud object storage.

Default is disabled ACL synchronization.

--vhb

Provides support of S3 style virtual hosting of buckets such as URL to use them during the AFM to cloud object storage creation. For the --vhb option, an endpoint has the https://bucket-name.s3.region-code.amazonaws.com format. AFM still requires a region to add it to the URL and separate it by using ‘@’ symbol.

For example,

1. Add a key.

   # mmafmcoskeys regbkt1:ap-south-1@regbkt1.s3.amazonaws.com set akey1 skey1

2. Verify whether the key was added successfully.

   # mmafmcoskeys regbkt1:ap-south-1@regbkt1.s3.amazonaws.com get

   A sample output is as follows:

   regbkt1:ap-south-1@regbkt1.s3.amazonaws.com=COS:akey1:skey1

3. Create an AFM to cloud object storage fileset.

   # mmafmcosconfig fs1 iw1  --endpoint https://ap-south-1@regbkt1.s3.amazonaws.com --object-fs --xattr --bucket regbkt1 --mode iw --acls --vhb --directory-object

   Note: To enable AFM to cloud object storage for standard other S3 cloud object storage backends, above options are not needed. Default option implicitly enables AFM to use standard S3 APIs.

--gcs

The --gcs parameter is used for Google cloud object storage backend. Enabling this parameter makes AFM to cloud object storage compatible with the list operations used by the Google cloud object.

--azure

The --azure parameter is used for Azure Blob Storage backend. This parameter must be specified to use Azure Blob Storage.

--prefix Prefix

Specifies a prefix to organize data inside a bucket where AFM can synchronize all the fileset data to the prefix. After an AFM to cloud object storage fileset is created by specifying the prefix, AFM appends the prefix to the bucket and use this root path as the target path. AFM will now upload or download all the fileset data to this prefix path under the bucket.

--directory-object

AFM to cloud object storage supports directory objects. All directories with and without objects can now be synchronized to the cloud object storage. You can now set extended attributes on directories with the directory object support.

--async-prefetch-interval

Specifies an interval when AFM automatically pulls the changed data and the metadata from a cloud object storage bucket to an AFM to cloud object storage fileset asynchronously.

When an interval is set, after the interval, AFM automatically synchronizes the changed data from the bucket to the AFM cache. the data and the metadata is downloaded areperiodically.

This option value can be set between 0 and 2,147,483,647. Default value is 30 minutes.

--verbose

Displays details of all operations that are performed during the creation of an AFM to cloud object fileset on the terminal. This parameter is optional.

--report

This option is used to generate a colon (':') separated information of the AFM to cloud object storage enabled fileset. This option shows information such as policy, quota that is set on the fileset. This option generates a report of the specified bucket and serverName combination.

To change the configuration or a parameter of an AFM to cloud object storage fileset that is already created, you need to run separate commands such as mmchfileset and mmchpolicy.

Exit status

Security

You must have root authority to run the mmafmcosconfig command.

The node on which the command is issued must be able to run remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.

See also

Location

/usr/lpp/mmfs/bin