Cloud data sharing

You can share data between storage servers by using the import and export function available in IBM Storage Scale.

Cloud data sharing works by combining the import and export functions that allow data to be moved across disparate geographical locations and/or heterogeneous application platforms. Cloud data sharing maintains a set of records of those moves called a manifest that enable applications to know what has moved. An application at one site can generate data, export it to the cloud, and applications at other sites can import and process that data. Applications can know what data has moved and is, therefore, now available by looking at the manifest file. It is also a way to easily move data back and forth between local and cloud storage systems. Cloud data sharing supports moving data to the cloud and pulling data from the cloud. Cloud data sharing must be configured with a local file system and a cloud account. Once configured, data can be moved between the IBM Storage Scale file system and the cloud account.

Application considerations

Exporting applications need some mechanism to both notify other applications that new data is available on the cloud and give those applications some way of understanding what objects were put to the cloud. Cloud data sharing services provide a manifest to help applications communicate that new data is available and what that data is. When data is exported, an option to build a manifest file can be specified. This manifest is a text file that contains the name of the cloud objects exported and some other information that can be used by an application that wants to import the full data, or a subsection of it.

When data is imported, there are cases in which not all the data is needed and this unneeded data can be identified by information in the file metadata. In these cases, it is recommended that as a first pass the file headers are imported only with the import-only-stub option. The policy engine can then be used to import only those files that are needed, thereby saving transfer time and cost. For now this import of stub includes metadata only for data that was previously exported by IBM Storage Scale.

Note: For many cloud services, enabling indexed containers can impact performance, so it is possible that cloud containers are not indexed. For these situations, a manifest is mandatory. But even with indexing enabled, for large containers that contain many objects, a manifest can be useful.

Additionally, this manifest utility can be used by a non-IBM Storage Scale application to build a manifest file for other applications, including IBM Storage Scale, to use for importing purposes.

There is a manifest utility that can run separate from IBM Storage Scale (it is a Python script) that can be used to look at the manifest. It provides a way to list and filter the manifest content, providing comma separated value output.

An overview of using import and export CLI commands

To export files to a cloud storage tier, issue a command according to the following syntax:


mmcloudgateway files export
    [--tag Tag ]
    [--target-name TargetName ]
    [--container Container | no-container ]
    [--manifest-file ManifestFile ]
    [--export-metadata [--fail-if-metadata-too-big ]]
    [--strip-filesystem-root ]
    File[ File ] }

The following example exports a local file named /dir1/dir2/file1 to the cloud and store it in a container named MyContainer. A manifest file will be created, and the object exported to the cloud will have an entry in that manifest file tagged with MRI_Images.


mmcloudgateway files export --container MyContainer --tag MRI_Images --export-metadata --manifest-file
/dir/ManifestFile /dir1/dir2/file1

To import files from a cloud storage tier, issue a command according to the following syntax:


mmcloudgateway files import
    [--container Container | no-container ]
    [--import-only-stub]
    [--import-metadata ]
    { [--directory Directory] | [--directory-root DirectoryRoot] | [--target-name TargetName] }
    { PolicyFile -e | [--] File[ File ] }

The following example imports files from the cloud storage tier and creates a necessary local directory structure.

mmcloudgateway files import --directory /localdir /dir1/dir2/file1

For more information on the usage of the import and export functions, see the mmcloudgateway man page.