Downloading objects

Download of an object is important when the AFM to cloud object storage relation is created in the ObjectOnly mode. With the ObjectOnly mode, the fileset does not automatically synchronize with a cloud object storage. This behavior is the default behavior of operation.

You need to manually download data or metadata from the object storage server into the AFM to cloud object storage filesets by using the mmafmcosctl command. The data transfer from a fileset to an object storage server does not need any manual intervention (single-writer (SW), independent-writer (IW) mode).

When selected objects are needed for an application to run on an AFM to cloud object storage fileset, they can be downloaded by using the --object-list option. All objects can be download by using the --all option. These objects are prefetched or downloaded from a cloud object storage.

Start of change You can specify individual objects and also directories containing objects in the same list-file and use this list-file for downloading them from the bucket. This list-file is a line separated file, where each entry is specified in a line. This list-file must not have any space in between two lines. End of change

The modifications to objects on a cloud object storage are not synchronized to the cache. Therefore, you need to download the objects. The modification to objects in IW and SW-modes is pushed to the cloud object storage.

When you download an object from AFM to cloud object storage or list objects, many objects or nested directories on a cloud object storage have extended attributes. The read directory operation takes time to populate metadata because the extended attributes need to be fetched.

In many AFM to cloud object storage use cases such as running analytics, when extended attributes (xattrs) synchronization is not required at the fileset level, set the afmObjectFastReaddir parameter value to yes for the improved read directory performance. To set this parameter, you must stop a fileset and start the fileset by using mmafmctl and mmchfileset commands.

When the afmObjectFastReaddir parameter value is set to yes, the AFM to cloud object storage read directory operation ingests metadata faster. Therefore, the metadata and later data read performance is improved. Extended attributes are not fetched from a cloud object storage. Also, objects that are deleted on a cloud object storage are not reflected in an AFM cache.

Failed Object list

While downloading objects, because of network errors some objects can fail to download. To resolve this problem, system administrators can use --enable-failed-file-list option from mmafmcosctl download command.

Enabling --enable-failed-file-list option creates a list of failed objects on the gateway node or node specified in the command. System administrators can run the download command again by using --object-list parameter and the failed file list is generated to download the objects again.

You can specify the -Y option to get the CSV output format. End of change

You can also specify non-existent numeric --uid and --gid options. End of change

Create an ObjectOnly IW-mode fileset.
```
mmafmcosconfig fs1 indwriter --endpoint  http://c1f1u11n07
 --uid 0 --gid 0 --new-bucket indwriter --mode iw 
```
You can also specify --uid, --gid as a nonexistent numeric UID or GID. You might require these UID or GID might for a containerized environment.

Check the fileset status.

mmlsfileset fs1 indwriter --afm -L

A sample output is as follows:

Filesets in file system 'fs1':

Attributes for fileset indwriter:
==================================
Status                                  Linked
Path                                    /gpfs/fs1/indwriter
Id                                      1
Root inode                              524291
Parent Id                               0
Created                                 Fri Oct 23 05:17:32 2020
Comment                                 
Inode space                             1      
Maximum number of inodes                100352
Allocated inodes                        100352
Permission change flag                  chmodAndSetacl
afm-associated                          Yes
Target                                  http://c1f1u11n07:80/indwriter
Mode                                    independent-writer
File Lookup Refresh Interval            disable
File Open Refresh Interval              disable
Dir Lookup Refresh Interval             disable
Dir Open Refresh Interval               disable
Async Delay                             15 (default)
Last pSnapId                            0
Display Home Snapshots                  no
Parallel Read Chunk Size                0
Number of Gateway Flush Threads         16
Prefetch Threshold                      0 (default)
Eviction Enabled                        yes (default)
Parallel Write Chunk Size               0
IO Flags                                0x400000 (afmSkipHomeRefresh)

Check whether you can write a file from the fileset.
```
touch /gpfs/fs1/indwriter/dd
```

Check whether the data is synchronized.

mmafmctl fs1 getstate

A sample output is as follows:

Fileset Name    Fileset Target                  Cache State   Gateway Node  Queue Length   Queue numExec 
------------    --------------                  ------------  -----------  ------------   ------------- 
indwriter       http://c1f1u11n07:80/indwriter  Dirty         c7f2n05       1              0

Check the fileset status.

mmafmctl fs1 getstate

A sample output is as follows:

Fileset Name    Fileset Target                  Cache State   Gateway Node  Queue Length   Queue numExec 
------------    --------------                  ------------  -----------   ------------   ------------- 
indwriter       http://c1f1u11n07:80/indwriter  Active         c7f2n05       0              1

The created object is synchronized.

Create objects on a cloud object storage.

Name      : dd
Date      : 2020-10-23 05:13:59 EDT 
Size      : 0 B    
ETag      : d41d8cd98f00b204e9800998ecf8427e 
Type      : file 
Metadata  :
  Content-Type: application/octet-stream 

Name      : obj1
Date      : 2020-10-23 05:20:48 EDT 
Size      : 2.2 KiB 
ETag      : 6fdfdb7af5eb5ad7f610014985951941 
Type      : file 
Metadata  :
  Content-Type: application/x-sh 

Name      : obj2
Date      : 2020-10-23 05:20:50 EDT 
Size      : 2.2 KiB 
ETag      : 6fdfdb7af5eb5ad7f610014985951941 
Type      : file 
Metadata  :
  Content-Type: application/x-sh 

Name      : obj3
Date      : 2020-10-23 05:20:51 EDT 
Size      : 2.2 KiB 
ETag      : 6fdfdb7af5eb5ad7f610014985951941 
Type      : file 
Metadata  :
  Content-Type: application/x-sh

Generate a list file of the created objects.

cat /listfile
/gpfs/fs1/indwriter/obj1
/gpfs/fs1/indwriter/obj2

Download the objects.

mmafmcosctl fs1 indwriter /gpfs/fs1/indwriter download  --object-list /listfile

A sample output is as follows:

     Queued     (Total)     Failed       AlreadyCached          TotalData
                                                        (approx in Bytes)
          2         (3)          0                  0                4544 
Object Download successfully queued at the gateway.

Ensure that data is downloaded.

ls -lash /gpfs/fs1/indwriter/

A sample output is as follows:

total 260K
 512 drwx------     5 root root 4.0K Oct 23 05:54 .
256K drwxr-xr-x     4 root root 256K Oct 23 05:18 ..
 512 drwx------ 65535 root root 4.0K Oct 23 05:17 .afm
 512 drwx------ 65535 root root 4.0K Oct 23 05:17 .pconflicts
 512 drwx------ 65535 root root 4.0K Oct 23 05:17 .ptrash
 512 dr-xr-xr-x     2 root root 8.0K Dec 31  1969 .snapshots
   0 -rw-r--r--     1 root root    0 Oct 23 05:19 dd
 512 -rwx------     1 root root 2.3K Oct 23  2020 obj1
 512 -rwx------     1 root root 2.3K Oct 23  2020 obj2

The objects are downloaded.