Improving write and remove operations efficiency in the manual updates mode

The manual updates (MU) mode fileset provides the flexibility to upload and download files or objects to and from cloud object storage after you finalize the set of objects to upload or download. Unlike other AFM to cloud object storage objectfs fileset modes, MU mode depends on manual intervention from administrators to upload and download the data to be in-sync. Administrators can also automate upload and download by using ILM policies to search specific files or objects to upload or download or remove.

When the workload consists of file creates and deletes, you can setup this features for deleting the files from other gateway nodes. After the files are deleted from AFM to cloud object storage MU-mode fileset, the files are not deleted from cloud object storage backends only by using the mmafmcosctl delete command. You can now use --gateway option to queue the deletes from target operations on a different gateway node.

The write and remove operations efficiency gives the following advantages:

Reduced replication delays: By separating queues for write class operations and remove operations, replication to cloud object storage will be more efficient, reducing delays and ensuring data consistency.
Improved system performance: Independent queuing can help optimize system resources and improve overall performance.
Scalability: This solution can handle high-volume create, write, delete workloads by ensuring that the system remains scalable and responsive.

Check the MU-mode fileset.

mmlsfileset fs1 mu1 -X

A sample output is as follows:

Filesets in file system 'fs1':
Attributes for fileset mu1:
============================
Status Linked
Path /gpfs/fs1/mu1
Id 5
Root inode 3145731
Parent Id 0
Created Mon Sep 23 04:43:27 2024
Comment
Inode space 3
Maximum number of inodes 100352
Allocated inodes 100352
Permission change flag chmodAndSetacl
IAM mode off
afm-associated Yes
Permission inherit flag inheritAclOnly
Target http://s3.us-east-1.amazonaws.com:80/scaleafmp
Mode manual-updates
File Lookup Refresh Interval 120
File Open Refresh Interval 120
Dir Lookup Refresh Interval 120
Dir Open Refresh Interval 120
Async Delay disable
Last pSnapId 0
Display Home Snapshots no
Parallel Read Chunk Size 0
Number of Gateway Flush Threads 32
Prefetch Threshold 0 (default)
Eviction Enabled yes (default)
IO Flags 0x8080000 (afmObjectXattr,afmObjectACL)
IO Flags2 0x0 (default)

Check whether the fileset contains any files.
```
ls -sh /gpfs/fs1/mu1
```
A sample output is as follows:
```
total 0
```

Create three files.

echo 12345 >> /gpfs/fs1/mu1/file1

echo 12345 >> /gpfs/fs1/mu1/file2

echo 12345 >> /gpfs/fs1/mu1/file3

Verify whether the files are created.

ls -l /gpfs/fs1/mu1/

A sample output is as follows:

total 0
-rw-r--r-- 1 root root 6 Oct 22 05:01 file1
-rw-r--r-- 1 root root 6 Oct 22 05:01 file2
-rw-r--r-- 1 root root 6 Oct 22 05:01 file3

Upload files to a cloud object storage.

mmafmcosctl fs1 mu1 /gpfs/fs1/mu1 upload --all

A sample output is as follows:

Queued Failed TotalData
(approx in Bytes)
3 0 18
Object Upload successfully queued at the gateway.

Check the gateway node.

getstate

A sample output is as follows:

Fileset Name Fileset Target                                 Cache State   Gateway Node Queue Length Queue numExec
------------ ---------------------------------------------- ------------- ------------ ------------ -------------
mu1          http://s3.us-east-1.amazonaws.com:80/scaleafmp Active        c7f2n02       0           3

Check whether the files are present on the cloud object storage.

aws console] aws ls aws/scaleafmp

A sample output is as follows:

[2024-10-22 04:58:17 EDT] 6B file1
[2024-10-22 04:58:17 EDT] 6B file2
[2024-10-22 04:58:17 EDT] 6B file3

Locally, remove the files from the MU fileset.
```
rm -rf /gpfs/fs1/mu1/f*
```
```
ls -sh /gpfs/fs1/mu1
```
A sample output is as follows:
```
total 0
```

Use the mmafmcosctl delete --gateway option to queue the files for removal from other gateway node.

mmafmcosctl fs1 mu1 /gpfs/fs1/mu1 delete --from-target --gateway c7f2n03

A sample output is as follows:

2024-10-22_04:59:28.797-0400: [N] AFM: Running the following policy scan to find remove operations from /gpfs/fs1/mu1.

Remove operations file list.

/var/mmfs/afm/fs1-5/recovery/removeOpsFile

Check that files are deleted from the cloud object storage.
```
aws console] aws ls aws/scaleafmp
```
A sample output is as follows:
```
No files present
```