AFM to COS upload and download statistics

You can get an ongoing statistics report when AFM to cloud object storage upload and download task is in progress.

AFM to cloud object storage fileset modes have upload and download features to support upload and download files to and from cloud object storage. AFM to cloud object storage filesets can be used in different modes of operations where manual upload and download of files is needed. After the files are uploaded or downloaded from COS mmafmcosctl command with –upload-stats or –download-stats can be used to see the progress of upload or download task respectively. The output of mmafmcosctl command shows number of files uploaded or downloaded as well as total data in bytes and its throughput.

Note:

In case of failed file download or upload, administrators can run mmafmctl checkUncached and checkDirty command to identify the files to download and upload again, respectively.

Examples of upload and download statistics:

Upload statistics
  1. Check the fileset.
    Filesets in file system 'fs1':
Name                     Status    Path                                       afmTarget                               
root                     Linked    /gpfs/fs1                                  --                                      
demo1bucket              Linked    /gpfs/fs1/demo1bucket                      http://s3.us-east.cloud-object-storage.appdomain.cloud:80/demo1bucket
  2. Create three files in fileset.
    Node1 ] dd if=/dev/urandom of=/gpfs/fs1/demo1bucket/file1 bs=256K count=40
40+0 records in
40+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0499466 s, 210 MB/s
Node1 ] dd if=/dev/urandom of=/gpfs/fs1/demo1bucket/file2 bs=256K count=40
40+0 records in
40+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0494641 s, 212 MB/s
Node1 ] dd if=/dev/urandom of=/gpfs/fs1/demo1bucket/file3 bs=256K count=40
40+0 records in
40+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0480755 s, 218 MB/s
Node1 ]
  3. Create uploadobjectlist file.
    Node1 ] cat  /root/uploadobjectlist
/gpfs/fs1/demo1bucket/file1
/gpfs/fs1/demo1bucket/file2
/gpfs/fs1/demo1bucket/file3
Node1 ]
  4. Upload these objects by using mmafmcosctl command.
    Node1 ] mmafmcosctl fs1 demo1bucket /gpfs/fs1/demo1bucket/ upload --object-list /root/uploadobjectlist
     Queued    (Total)      Failed                TotalData
                                              (approx in Bytes)
         3       (3)           0                  31457280 

Object Upload successfully queued at the gateway.
  5. check the upload progress.
    Node1 ] mmafmcosctl fs1 demo1bucket --upload-stats
mmafmcosctl: Statistics of last or currently running upload are as follows:
Fileset Name    Pending         Failed          Total           Total data(Bytes)    Throughput(KB/s)
------------    ------------    ------------    ------------    -------------------- ---------------
demo1bucket     2               0               3               10485760             5120            
Node1 

Node1 ] mmafmcosctl fs1 demo1bucket --upload-stats
mmafmcosctl: Statistics of last or currently running upload are as follows:
Fileset Name    Pending         Failed          Total           Total data(Bytes)    Throughput(KB/s)
------------    ------------    ------------    ------------    -------------------- ---------------
demo1bucket     0               0               3               31457280             0
Download statistics
  1. Files newfile1-3 are uploaded to COS.
    COScmline ] ls ibmcos/demo1bucket
[2022-01-21 06:15:40 EST] 10MiB	 file1
[2022-01-21 05:50:04 EST]  10MiB file2
[2022-01-21 05:50:01 EST]  10MiB file3
c7f2n03 21Jan06:43:00 0]
  2. Create an Object list for download.
    Node1 ] cat  /root/downloadobjectlist
/gpfs/fs1/demo1bucket/newfile1
/gpfs/fs1/demo1bucket/newfile2
/gpfs/fs1/demo1bucket/newfile3
Node1 ]
  3. Download the object.
    Node1 ] mmafmcosctl fs1 demo1bucket /gpfs/fs1/demo1bucket/ download --object-list /root/downloadobjectlist
     Queued     (Total)     Failed       AlreadyCached          TotalData
                                                        (approx in Bytes)
          3         (3)          0                  0            31457280 
Object Download successfully queued at the gateway.
Node1 :59:52 2]
  4. Issue the following command to track download progress:
    Node1 :59:53 2] mmafmcosctl fs1 demo1bucket --download-stats
mmafmcosctl: Statistics of last or currently running download are as follows:
Fileset Name    Pending         Failed          Already cached  Total           Total data(Bytes)    Throughput(KB/s)
------------    ------------    ------------    --------------- ------------    -------------------- ---------------
demo1bucket     0               0               0               3               31457280             0
  5. List the files in the fileset to confirm the download.
    Node1] ls /gpfs/fs1/demo1bucket/
file1  file2  file3 newfile1  newfile2  newfile3