Creating AFM to cloud object storage relation in different modes

You can set an IBM Storage Scale cluster to create AFM to cloud object storage filesets to connect to cloud object services such as Amazon S3 Start of change , Microsoft Azure Blob, End of change and IBM Cloud® Object Storage. To set an AFM to cloud object storage relation, you can use read-only (RO), single-writer (SW), local-update (LU), and independent-writer (IW) modes in the AFM to cloud object storage.

When you create an AFM to cloud object storage relation by using the mmafmcosconfig command, use the --mode option to define the mode of operation for the relation.

Read-only (RO)

In this mode, data in an AFM to cloud object storage fileset is read-only. You cannot create or modify objects in the fileset. If an application uses an object while it is being re-created (deleted and re-created with the same name) on a cloud object storage server, it is re-created in the cache. When the RO-mode fileset is set, download or prefetch can bring the data of necessary objects, or objects can be brought from the cloud object storage to the fileset on-demand while objects are being read.

Note: When an RO-mode AFM to cloud object storage relation is set, any metadata operation such as ls populates the fileset with metadata.

Example:

Create an RO-mode AFM to cloud object storage relation.

mmafmcosconfig fs1 readonly  --endpoint  http://c1f1u11n07
--uid 0 --gid 0 --new-bucket readonly --mode ro --cleanup  --object-fs

Determine the RO-mode of the fileset.

mmlsfileset fs1 readonly --afm -L

A sample output is as follows:

Filesets in file system 'fs1':
Attributes for fileset readonly:
=================================
Status                                  Linked
Path                                    /gpfs/fs1/readonly
Id                                      1
Root inode                              524291
Parent Id                               0
Created                                 Tue Oct 20 08:53:44 2020
Comment                                 
Inode space                             1      
Maximum number of inodes                100352
Allocated inodes                        100352
Permission change flag                  chmodAndSetacl
afm-associated                          Yes
Target                                  http://c1f1u11n07:80/readonly
Mode                                    read-only
File Lookup Refresh Interval            120
File Open Refresh Interval              120
Dir Lookup Refresh Interval             120
Dir Open Refresh Interval               120
Async Delay                             disable
Expiration Timeout                      disable (default)
Last pSnapId                            0
Display Home Snapshots                  yes (default)
Parallel Read Chunk Size                0
Number of Gateway Flush Threads         16
Prefetch Threshold                      0 (default)
Eviction Enabled                        yes (default)
IO Flags                                0x0 (default)

Create obj1, obj2, and obj3 objects on a cloud object storage.

Check whether the objects are cached.

ls -lsh /gpfs/fs1/readonly/

A sample output is as follows:

total 0
0 -rwx------ 1 root root 13 Oct 20  2020 obj1
0 -rwx------ 1 root root 13 Oct 20  2020 obj2
0 -rwx------ 1 root root 13 Oct 20  2020 obj3

Pull objects data from the cloud object storage to the AFM to cloud object storage fileset.
```
cat /gpfs/fs1/readonly/obj1
```
A sample output is as follows:
```
111111111111
```
```
cat /gpfs/fs1/readonly/obj2
```
A sample output is as follows:
```
111111111111
```

Check the fileset status.

ls -lsh /gpfs/fs1/readonly/

A sample output is as follows:

total 1.0K
512 -rwx------ 1 root root 13 Oct 20  2020 obj1
512 -rwx------ 1 root root 13 Oct 20  2020 obj2
  0 -rwx------ 1 root root 13 Oct 20  2020 obj3

Note: Data of the obj3 object is not read. Therefore, the object is uncached in the fileset.

Single-writer (SW)

In this mode, only an AFM to cloud object storage fileset can do all the write operation or data is generated on the SW-mode fileset and the fileset does not check the cloud object storage for file or object updates. Ensure that no write operation is done on the cloud object storage server.

Note: You cannot enforce this check.

A cloud object storage server can have some pre-existing data. The SW-mode fileset can replicate this data by using the download or prefetch operation, and this data can be updated. Any updates in the fileset data are queued on the gateway node and passed to the cloud object storage server. For more information about the download operation, see mmafmcosctl command.

Example:

Create an SW-mode AFM to cloud object storage relation.

mmafmcosconfig fs1 singlewriter --endpoint  http://c1f1u11n07
--uid 0 --gid 0 --new-bucket singlewriter  --mode sw --cleanup  --object-fs

Check whether the objects are cached.
```
ls -lsh /gpfs/fs1/singlewriter/
```
A sample output is as follows:
```
total 0
```

Create the objects in an IBM Storage Scale cluster.

for a in `seq 3`;  do  dd if=/dev/urandom of=/gpfs/fs1/singlewriter/object$a  count=12 bs=256K;  done

A sample output is as follows:

12+0 records in
12+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0155518 s, 202 MB/s
12+0 records in
12+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0150043 s, 210 MB/s
12+0 records in
12+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0160235 s, 196 MB/s

Check the cache state.

mmafmctl fs1 getstate

A sample output is as follows:

Fileset Name  Fileset Target                     Cache State  Gateway Node Queue Length Queue numExec 
------------  --------------                     -----------  -----------  ------------ ------------- 
singlewriter  http://c1f1u11n07:80/singlewriter  Active         c7f2n05       0           7

Check whether the objects are synchronized to a cloud object storage server from a cloud object storage GUI.

Name      : object1
Date      : 2020-10-20 08:54:06 EDT 
Size      : 3.0 MiB 
ETag      : b7a173514d704481128f96bae96c4735 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 

Name      : object2
Date      : 2020-10-20 08:54:07 EDT 
Size      : 3.0 MiB 
ETag      : d983316f3457644c45f3db63fb496060 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 
 
Name      : object3
Date      : 2020-10-20 08:54:07 EDT 
Size      : 3.0 MiB 
ETag      : 862841fbc08eff03878230a0b32e7c71 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream

Independent-writer (IW)

This mode allows multiple AFM to cloud object storage filesets to point to the same cloud object storage bucket. Multiple AFM to cloud object storage filesets can be on the same IBM Storage Scale cluster or on a different cluster. Also, they point to the cloud object storage server. There is no synchronous locking between clusters when objects are updated on the cloud object storage server. Each AFM to cloud object storage fileset reads from the cloud object storage server and makes updates to the cloud object storage server independently. Reads and updates are based on the revalidation intervals and the asynchronous delay.

This mode is used to access different objects from each IW-mode AFM to cloud object storage. For example, unique users on each site are updating objects in their cloud object storage server bucket. Although this mode allows multiple AFM to cloud object storage clusters to modify the same objects, only advanced users must modify the objects because there is no locking or ordering between updates. Updates are propagated to the cloud object storage server in an asynchronous manner and can be delayed because of network disconnections. Therefore, conflicting updates from multiple AFM to cloud object storage sites can cause the data on the cloud object storage server to be undetermined.

Example:

Create an IW-mode AFM to cloud object storage relation.

mmafmcosconfig fs1 indwriter --endpoint  http://c1f1u11n07 
--uid 0 --gid 0  --new-bucket indwriter --mode iw --cleanup  --object-fs

Check whether objects are cached.
```
ls -lsh /gpfs/fs1/indwriter/
```
A sample output is as follows:
```
total 0
```

Create objects in an IBM Storage Scale cluster.

for a in `seq 3`;  do  dd if=/dev/urandom of=/gpfs/fs1/indwriter/object$a  count=12 bs=256K;  done

A sample output is as follows:

12+0 records in
12+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0158949 s, 198 MB/s
12+0 records in
12+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0150224 s, 209 MB/s
12+0 records in
12+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0153003 s, 206 MB/s

Check the fileset status.

ls -lsh /gpfs/fs1/indwriter/

A sample output is as follows:

total 9.0M
3.0M -rw-r--r-- 1 root root 3.0M Oct 20 09:04 object1
3.0M -rw-r--r-- 1 root root 3.0M Oct 20 09:04 object2
3.0M -rw-r--r-- 1 root root 3.0M Oct 20 09:04 object3

On COS:
Name      : object1
Date      : 2020-10-20 08:59:04 EDT 
Size      : 3.0 MiB 
ETag      : a1e25de2378c86479323de2345422923 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 

Name      : object2
Date      : 2020-10-20 08:59:05 EDT 
Size      : 3.0 MiB 
ETag      : 1bfa4345ba48dffdacc7037ce57cb112 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 
Name      : object3
Date      : 2020-10-20 08:59:05 EDT 
Size      : 3.0 MiB 
ETag      : c8d7c7a1da270b2ab5d2675083842326 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream

Create obj1, obj2, and obj3 objects.

Check whether the objects are cached.

ls -lsh /gpfs/fs1/indwriter/

A sample output is as follows:

total 0
0 -rwx------ 1 root root 13 Oct 20  2020 obj1
0 -rwx------ 1 root root 13 Oct 20  2020 obj2
0 -rwx------ 1 root root 13 Oct 20  2020 obj3

Pull objects data from the cloud object storage to the AFM to cloud object storage fileset.
```
cat /gpfs/fs1/indwriter/obj1
```
A sample output is as follows:
```
111111111111
```
```
cat /gpfs/fs1/indwriter/obj2
```
A sample output is as follows:
```
111111111111
```

Check whether the objects are cached.

ls -lsh /gpfs/fs1/indwriter/

A sample output as follows:

total 1.0K
512 -rwx------ 1 root root 13 Oct 20  2020 obj1
512 -rwx------ 1 root root 13 Oct 20  2020 obj2
  0 -rwx------ 1 root root 13 Oct 20  2020 obj3

Local updates (LU)

The LU-mode behaves in a similar way as the RO mode. However, you can create and modify objects in the AFM to cloud object storage fileset. Updates in the fileset are considered local to the AFM to cloud object storage and are decoupled from the corresponding object on the cloud object storage server. Local updates are never pushed back to the cloud object storage server. When an object is modified, during the revalidation operation, the object is not compared to the version on the cloud object storage server to verify whether it is up to date. Changes of this object on the cloud object storage server do not have an impact on the replicated copy of the object and the object on the cloud object storage server.

Behaviors with local objects:

In AFM to cloud object storage, LU-mode objects have one of the following states:

Uncached: Objects on the cloud object storage server are shown in the AFM to cloud object storage as uncached. For these objects, only metadata is copied into the AFM to cloud object storage. The object does not reside on the AFM to cloud object storage, but only on the cloud object storage server. Changes on the cloud object storage server are reflected in the AFM to cloud object storage.

Cached or replicated: If an uncached object is read in the AFM to cloud object storage or pre-fetched, the state of the object changes to replicated or cached. In the replicated state, all changes to the object on the cloud object storage server are reflected in the AFM to cloud object storage. The object resides on the AFM to cloud object storage.

Local: Object data or metadata that is modified on AFM to cloud object storage becomes local to the AFM to cloud object storage. The replicated objects relationship to the object in the cloud object storage server is broken. Changes on the cloud object storage server are not reflected in the AFM to cloud object storage and object changes are not copied to the cloud object storage server.

Note: Objects can be downloaded from a cloud object storage server and uploaded back without affecting the LU-mode semantics. When the object is downloaded, the object is synchronized to a cloud object storage object.

Example:

Create an LU-mode AFM to cloud object storage relation.

mmafmcosconfig fs1 localupdates --endpoint  http://c1f1u11n07
 --uid 0 --gid 0 --new-bucket localupdates --mode lu --cleanup  --object-fs

Create three objects that have per-existing data on a cloud object storage.

Check whether the objects are cached.

ls -lsh /gpfs/fs1/localupdates

A sample output is as follows:

total 0
0 -rwx------ 1 root root 13 Oct 20  2020 obj1
0 -rwx------ 1 root root 13 Oct 20  2020 obj2
0 -rwx------ 1 root root 13 Oct 20  2020 obj3

Pull objects data from the cloud object storage to the AFM to cloud object storage fileset.
```
cat /gpfs/fs1/localupdates/obj1
```
A sample output is as follows:
```
111111111111
```
```
cat /gpfs/fs1/localupdates/obj2
```
A sample output is as follows:
```
111111111111
```
```
cat /gpfs/fs1/localupdates/obj3
```
A sample output is as follows:
```
111111111111
```

Check the fileset status.

ls -lsh /gpfs/fs1/localupdates

A sample output is as follows:

total 1.5K
512 -rwx------ 1 root root 13 Oct 20  2020 obj1
512 -rwx------ 1 root root 13 Oct 20  2020 obj2
512 -rwx------ 1 root root 13 Oct 20  2020 obj3

Check the modified the contents.

echo 2222222 >> /gpfs/fs1/localupdates/obj1
echo 2222222 >> /gpfs/fs1/localupdates/obj2
echo 2222222 >> /gpfs/fs1/localupdates/obj3

A sample output is as follows:

Name      : obj1
Date      : 2020-10-20 09:10:39 EDT 
Size      : 13 B   
ETag      : 87b8769b874865e65a4525bfe9e56ba8 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 

Name      : obj2
Date      : 2020-10-20 09:10:44 EDT 
Size      : 13 B   
ETag      : 87b8769b874865e65a4525bfe9e56ba8 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 

Name      : obj3
Date      : 2020-10-20 09:10:49 EDT 
Size      : 13 B   
ETag      : 87b8769b874865e65a4525bfe9e56ba8 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream

To push the created objects to the cloud object storage, upload the objects.

mmafmcosctl fs1 localupdates /gpfs/fs1/localupdates upload --all

A sample output is as follows:

     Queued         Failed                TotalData
                                  (approx in Bytes)
         3               0                       63 
Object Upload successfully queued at the gateway.

on COS : 
Name      : obj1
Date      : 2020-10-20 09:19:08 EDT 
Size      : 21 B   
ETag      : a29344969f1524d72a050e910bb20ab0 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 
 
Name      : obj2
Date      : 2020-10-20 09:19:08 EDT 
Size      : 21 B   
ETag      : a29344969f1524d72a050e910bb20ab0 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream

Name      : obj3
Date      : 2020-10-20 09:19:08 EDT 
Size      : 21 B   
ETag      : a29344969f1524d72a050e910bb20ab0 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream

Note: http://c1f1u11n07:80 cloud object storage endpoint is used in the relation examples.

Along with these modes of operations, the AFM to cloud object storage has two more behavioral modes. These modes are Object-FS and ObjectOnly.

Object-FS

In the Object-FS mode AFM to cloud object storage fileset is synchronized to the cloud object storage. RO, LU, and IW modes of filesets synchronize metadata to and from the cloud object storage server. It includes operations such as readdir and lookups for the synchronization. Objects are downloaded when they are read on-demand or an application that is running on fileset is working on it. The ObjectFS mode-enabled fileset behaves in a similar way as an AFM fileset. For SW and IW modes-enabled fileset, the AFM to cloud object storage uploads files as objects to the cloud object storage server. For AFM RO, LU, and IW mode-enabled fileset, the AFM to cloud object storage automatically synchronizes objects from the cloud object storage server to the fileset as files. Enable this parameter if the AFM to cloud object storage fileset needs to behave in a similar way as an AFM mode fileset.

ObjectOnly

With the ObjectOnly mode, the fileset does not automatically synchronize with cloud object storage server. This behavior is the default behavior of operation. You need to manually download the data or metadata from the cloud object storage server into the AFM to cloud object storage filesets by using the mmafmcosctl command. You can download the data or metadata while data is being transferred from the AFM to cloud object storage fileset to the cloud object storage server without any manual intervention (SW, IW mode).

In ObjectOnly mode, the performance is higher than ObjectFS mode because all file system operations are not enabled. As users traverse the directory or object tree of a fileset, the object information from the cloud object storage server is checked and updated as needed on the fileset.

Note:

You can also specify --uid, --gid as a nonexistent numeric UID or GID. You might require these UID or GID might for a containerized environment. End of change

AFM to cloud object storage operations are serviced either synchronously or asynchronously. Reads and revalidations are synchronous operations. Update operations from the fileset are asynchronous to cloud object storage server based on the afmAsyncDelay interval. All update operations from the writable filesets such as IW or SW mode are on the primary gateway. Queues in the primary gateway are pushed to cloud object storage server asynchronously based on the afmAsyncDelay interval.

Note: With the ObjectFS mode, objects can be readily read on-demand from the cloud object storage server. Whereas the ObjectOnly mode download and upload operations can be used for priority data synchronization based on the mode of AFM to cloud object storage fileset.