Downloading objects by using the outband method

The outband download method is an enhanced method for downloading files or objects into AFM to cloud object storage filesets.

The outband download method uses multiple threads and direct mmafmtransfer interface, which negates the effect of serializing operations by VFS layer for high latency networks and increases the overall performance for download. In this method, any node that can connect to cloud object storage can be used to download the files.

Multiple download instances can be run on multiple filesets by using multiple nodes that can connect to cloud object storage buckets.

When selected objects are needed for an application to run on an AFM to cloud object storage fileset, they can be downloaded by using the --object-list option. All objects can be download by using the --all option. These objects are prefetched or downloaded from a cloud object storage.

The outband download command supports the failed object. When some objects are not downloaded because of network error or other issue, the outband download command generates a list of those objects. An administrator can use this object list and download the objects again. Use --enable-failed-object-list options to enable this feature.

Start of changeIf you specify the --enable-failed-object-list option, AFM creates a failed object list of all the objects, which are not downloaded in the cache because of any error. AFM creates a file in the .afm AFM fileset directory, that is, /gpfs/fs1/fset-ro/.afm/.mmafmcosctl_failedQueuingList. You can check this file path to know which objects are not downloaded when the download operation was run to fetch objects again.End of change

Note:
  • When outband download method is used to download objects, an administrator can see the download progress, but queue is not built on gateway nodes and fileset can remain inactive. As gateway node queue is not built for outband download, the download stats that use command mmafmcosctl are not supported.
  • When -N nongateway node is specified for download, though download progresses, the command does not display continuous stats. On gateway node, command progression is displayed.
  • Outband download does not support mapped directory download and parallel I/O download. Read sizes from applications are ignored. Default part size is 16 M with eight number of parts are downloaded in parallel.
  • --directory-object option must be enabled while creating AFM to cloud object storage relations for outband download.
  • The object updated on the Cloud Object Storage bucket after the download can be downloaded again by using the outband download method.
Example
Bucket ibmc1 hosted on the IBM Cloud already has objects:
  1. To create a new fileset, issue the following command:
    mmafmcosconfig fs1 ibmc2  --endpoint  https://s3.us-east.cloud-object-storage.appdomain.cloud --object-fs   --bucket ibmc1 --mode ro --directory-object  --fast-readdir
  2. To download the object by using outband method, issue the following command:
    mmafmcosctl fs1 ibmc2  /gpfs/fs1/ibmc2 download  --all --outband  --threads 8 --enable-failed-object-list

    A sample output is as follows:

    
Queued          (Total)         Failed         AlreadyCached          TotalData (Bytes)
 ------          -------         ------         -------------          -----------------
  7800            11001             0                    0                  70890000
  11001           11001             0                    0                  100000000

    Download command completed.