Uploading and downloading files from MU mode fileset

The files can be uploaded and downloaded from cloud object storage by using AFM Manual Update (MU) mode.

MU mode fileset depends on mmafmcosctl upload and download commands to upload and download data to and from cloud object storage respectively.

Administrators can determine the files to upload and download from cloud object storage (COS). Files on AFM to cloud object storage MU fileset can be determined manually or by using policies and can be uploaded to the cloud object storage. Whereas, MU mode is not capable of automatically recognizing the new files that are added on COS by third-party interface or API. Therefore, administrators need to determine these files to download them into the MU fileset. To create and manage policies, see Policies for automating file management.

For upload and download, mmafmcosctl upload/download –object-list command needs to be used. The –object list can be created by using the absolute path of the objects with respect to the MU fileset.

Note: Use mmafmcosctl upload –all option of command with caution as this command with --all option uploads all the data to cloud object storage, and can overwrite the files on COS.
Example 1: Upload
  1. Create a new MU mode relation with COS bucket:
    1. Set keys from cloud for the new bucket.
      Node1]  mmafmcoskeys demobucket-mu:s3.us-east.cloud-object-storage.appdomain.cloud set 779c2178d4key497de8d6cdc72 ef08ac5bcc1c79f4key5a293a7fe24c2ef
fe54d2beba0
Node1]
    2. Create MU fileset.
      Node1] mmafmcosconfig fs1 demo_mu  --endpoint http://s3.us-east.cloud-object-storage.appdomain.cloud --new-bucket demobucket-mu --object-fs --mode mu
  2. Create files in MU mode fileset by using dd.
    
Node1] dd if=/dev/urandom of=/gpfs/fs1/demo_mu/file1 bs=256K count=4 
4+0 records in
4+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110485 s, 94.9 MB/s
Node1] dd if=/dev/urandom of=/gpfs/fs1/demo_mu/file2 bs=256K count=8
8+0 records in
8+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0124402 s, 169 MB/s
Node1] dd if=/dev/urandom of=/gpfs/fs1/demo_mu/file3 bs=256K count=12
12+0 records in
12+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0153719 s, 205 MB/s
Node1] 
Node1] dd if=/dev/urandom of=/gpfs/fs1/demo_mu/file4 bs=256K count=16
16+0 records in
16+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0223167 s, 188 MB/s
Node1] dd if=/dev/urandom of=/gpfs/fs1/demo_mu/file5 bs=256K count=20
20+0 records in
20+0 records out
5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0268569 s, 195 MB/s
Node1]
  3. List the files in MU fileset.
    
Node1] ls -shl /gpfs/fs1/demo_mu/
total 14M
1.0M -rw-r--r-- 1 root root 1.0M Jan 12 03:28 file1
2.0M -rw-r--r-- 1 root root 2.0M Jan 12 03:28 file2
3.0M -rw-r--r-- 1 root root 3.0M Jan 12 03:29 file3
4.0M -rw-r--r-- 1 root root 4.0M Jan 12 04:30 file4
4.0M -rw-r--r-- 1 root root 5.0M Jan 12 04:30 file5
Node1]
  4. Now upload the specific files to COS by using upload command object-list.
    1. Create an object list. The files determined to upload are file1, file3, and file5 as shown in the following sample:
      Node1] cat /root/objectlist1
/gpfs/fs1/demo_mu/file1
/gpfs/fs1/demo_mu/file3
/gpfs/fs1/demo_mu/file5
Node1]
    2. Issue upload command by using the following object list:
      Node1] mmafmcosctl fs1 demo_mu /gpfs/fs1/demo_mu/ upload --object-list /root/objectlist1
     Queued    (Total)      Failed                TotalData
                                              (approx in Bytes)
       3         (3)           0                  9437184
      Object Upload successfully queued at the gateway.
    3. Issue the following command to check the upload stats:
      Node1]  mmafmcosctl fs1 demo_mu --upload-stats
      
mmafmcosctl: Statistics of last or currently running upload are as follows:
Fileset Name    Pending         Failed          Total           Total data(Bytes)    Throughput(KB/s)
------------    ------------    ------------    ------------    -------------------- ---------------
demo_mu         0               0               3               9437184              0              
Node1 12Jan04:58:46 4]
  5. Check that the files are pushed to COS by using COS CLI. The following files are transferred to COS by using COS command-line tool as shown in the sample output:
    
[2022-01-12 04:58:16 EST] 1.0MiB file1
[2022-01-12 04:58:16 EST] 3.0MiB file3
[2022-01-12 04:58:16 EST] 5.0MiB file5
Example 2: Download
Note:
  • Manual updates mode fileset does not expect file data changes on the cloud object storage. When the file is downloaded from COS the file is marked as cached and then the file is not downloaded again.
  • For downloading files from COS to MU mode, --object-list must be used. Download --all option is not relevant to MU mode as this mode is not aware of the data that are created on the COS by the third-party services.
For download example, a MU fileset relation is created from existing bucket that contains existing data.
Bucket name - download-bucket

S3 commandline]  ls ibmcos/download-bucket
[2022-01-12 05:32:44 EST] 1.0MiB file1
[2022-01-12 05:32:52 EST] 2.0MiB file2
[2022-01-12 05:33:00 EST] 3.0MiB file3
[2022-01-12 05:33:07 EST] 4.0MiB file4
[2022-01-12 05:33:13 EST] 5.0MiB file5
S3 commandline]
  1. Create MU fileset with existing bucket.
    1. Setup keys for this bucket.
      node1]mmafmcoskeys download-bucket:s3.us-east.cloud-object-storage.appdomain.cloud set 7aff636483224d5keyf0c39 8268e298b91dkeyb8cb61e215bc288
078418743fe7e6ad
node1]
    2. Create MU mode fileset.
      node1] mmafmcosconfig fs1 download-mu  --endpoint http://s3.us-east.cloud-object-storage.appdomain.cloud --bucket  download-bucket  --object-fs --mode mu
  2. List files in MU fileset.
    
node1] ls -lsh /gpfs/fs1/download-mu/
total 0
node1]
  3. Create an object list referencing from which files to download from COS.
    
node1]  cat /root/objectlist1
/gpfs/fs1/download-mu/file1
/gpfs/fs1/download-mu/file3
/gpfs/fs1/download-mu/file5
node1 12Jan05:42:58 4]
  4. Download the files by using objectlist.
    node1] mmafmcosctl fs1 download-mu /gpfs/fs1/download-mu/ download --object-list /root/objectlist1
     Queued     (Total)     Failed       AlreadyCached          TotalData
                                                            (approx in Bytes)
          3         (3)          0                  0             9437184 
Object Download successfully queued at the gateway.
node1]
  5. Check download stats.
    node1 ] mmafmcosctl fs1 download-mu --download-stats
mmafmcosctl: Statistics of last or currently running download are as follows:
Fileset Name    Pending         Failed          Already cached  Total           Total data(Bytes)    Throughput(KB/s)
------------    ------------    ------------    --------------- ------------    -------------------- ---------------
download-mu     0               0               0               3               9437184              0
  6. Check that these files are downloaded in the MU mode fileset.
    
node1 ] ls -lsh /gpfs/fs1/download-mu/
total 9.0M
1.0M -rwxrwx--- 1 root root 1.0M Jan 12  2022 file1
3.0M -rwxrwx--- 1 root root 3.0M Jan 12  2022 file3
5.0M -rwxrwx--- 1 root root 5.0M Jan 12  2022 file5