Configuring asynchronous auto prefetch

The changed object data in a bucket on the cloud object storage must be synchronized to the cache. You can configure an AFM cloud object storage fileset in the read-only (RO) or independent writer (IW) mode to periodically fetch changed data asynchronously from a bucket on a cloud object storage.

The asynchronous prefetch interval synchronizes the created, modified, or deleted objects. This synchronization must be fast with the afmFastObjectReadDir2 parameter to avoid fetch ACLs and xattrs associated to the objects. After these parameters are configured, AFM updates the AFM cache fileset faster with changed data from the cloud object storage bucket.

Set an access key and a secret key for a bucket.
```
mmafmcoskeys bucket1:s3.us-south.cloud-object-storage.appdomain.cloud set akey skey
```
where bucket1 is a bucket for which an access key and a secret key need to be set.

Create an AFM to cloud object storage fileset in the RO or IW mode by using a bucket on the cloud object storage as a target.

mmafmcosconfig  fs1 ro1  --endpoint  https://s3.us-south.cloud-object-storage.appdomain.cloud 
--uid 0 --gid 0 --new-bucket bucket1 --mode ro --cleanup  --object-fs --directory-object 
--fast-readdir2 --async-prefetch-interval 30 –makeactive --verbose

In this command, a fileset is created in the RO mode.

Configure watch folder events for the created AFM fileset.

mmwatch fs1 enable --fileset ro1 --events IN_CREATE,IN_DELETE,IN_MODIFY --event-handler kafkasink --sink-topic ro1 --sink-brokers localhost:9092

Check active watch folders on the fileset.

mmwatch fs1 list

A sample output is as follows:

Filesystem fs1 has 3 watcher(s):
Cluster ID            WatchID/PID  Type Start Time               Path Description
---------------------------------------------------------------------------------------------------------------------- 
10258389552690212634 CLW1741845048 INODE Wed Mar 12 22-50-50 2025 /gpfs/fs1/ro1

For more information about the watch folder, see Clustered watch folder.

Start the ZooKeeper.

bin/zookeeper-server-start.sh config/zookeeper.properties

Start the Kafka Producer.

bin/kafka-server-start.sh config/server.properties

Start the Kafka Consumer.

bin/kafka-console-consumer.sh --topic ro1 --from-beginning --bootstrap-server localhost:9092

Create or modify some data in the cloud object bucket.

AFM fetches modified data from the home to the cache fileset. Watch folder events are generated and passed to the Kafka consumer.

Create some data in the cloud object bucket.

{
  "WF_JSON": "0.0.3",
  "wd": "1",
  "cookie": "0",
  "mask": "0x100",
  "event": "IN_CREATE",
  "path": "/gpfs/fs2/ro2/trcrpt.2025-03-03_09.36.24.866643.san1-sanpod1.gz",
  "clusterName": "new.fyre.ibm.com",
  "nodeName": "host1",
  "nfsClientIp": "",
  "fsName": "fs2",
  "inode": "2098181",
  "fileSetID": "3",
  "linkCount": "1",
  "openFlags": "0",
  "poolName": "system",
  "fileSize": "0",
  "ownerUserId": "0",
  "ownerGroupId": "0",
  "atime": "2025-04-02_00:07:32.735-0700",
  "ctime": "2025-04-02_00:07:32.735-0700",
  "mtime": "2025-04-02_00:07:32.735-0700",
  "eventTime": "2025-04-02_00:07:32.735-0700",
  "clientUserId": "0",
  "clientGroupId": "0",
  "accessMode": null,
  "processId": "NA",
  "bytesRead": null,
  "bytesWritten": null,
  "minReadOffset": null,
  "maxReadOffset": null,
  "minWriteOffset": null,
  "maxWriteOffset": null,
  "permissions": "200100770",
  "acls": null,
  "xattrs": null,
  "subEvent": "NONE"
}

Modify some data in the cloud object bucket.

{
  "WF_JSON": "0.0.3",
  "wd": "1",
  "cookie": "0",
  "mask": "0x2",
  "event": "IN_MODIFY",
  "path": "/gpfs/fs1/ro1/file2.host1.gz",
  "clusterName": "newcluster1.ibm.com",
  "nodeName": "host1",
  "nfsClientIp": "",
  "fsName": "fs1",
  "inode": "2098180",
  "fileSetID": "3",
  "linkCount": "1",
  "openFlags": "32771",
  "poolName": "system",
  "fileSize": "220760979",
  "ownerUserId": "0",
  "ownerGroupId": "0",
  "atime": "2025-04-02_00:07:32.729-0700",
  "ctime": "2025-04-02_00:07:32.779-0700",
  "mtime": "2025-04-02_00:06:56.000-0700",
  "eventTime": "2025-04-02_00:08:15.825-0700",
  "clientUserId": "0",
  "clientGroupId": "0",
  "accessMode": null,
  "processId": "442952",
  "bytesRead": "0",
  "bytesWritten": "220760979",
  "minReadOffset": "9223372036854775807",
  "maxReadOffset": "0",
  "minWriteOffset": "0",
  "maxWriteOffset": "220760978",
  "permissions": "200102770",
  "acls": null,
  "xattrs": null,
  "subEvent": "NONE"
}