Using IBM Cloud Object Storage retention enabled vaults
Nils Haustein 060000NX38 Visits (7534)
By Nils Haustein and Harald Seipp
In this blog article we explain the basic concepts of IBM
Object storage is the primary data storage provided in the cloud, and it is also increasingly used for on-premises solutions. Object storage is growing for the following reasons:
Due to its characteristics, object storage is becoming a significant storage repository for active archive of unstructured data, both for public and private clouds. Some types of archived data have to be kept in a writ
Introduction to IBM Cloud Object Storage
IBM COS can be accessed through an object interface using HTTP/REST Application Programming Interfaces (API), such as Amazon Web Services Simple Storage Service (AWS S3), OpenStack Swiftand the IBM Cloud Object Storage System™ Simple Object over HTTP (SOH) API. Object storage API operations - such as PUT, GET, DELETE and LIST - allow applications to manage objects. The objects are stored in a logical storage construct that is called a Vault. In the world of Amazon Web Services (AWS) a vault corresponds to a bucket.
Starting with IBM COS level 3.12 (released in November 2017), vaults can be configured in compliance mode. A vault configured in compliance mode is also called retention enabled vault or retention vault. To learn more about the basic concepts, architectures and advantages of IBM COS have a look at the corresponding IBM Redbook publications  and .
IBM COS retention enabled vaults
An IBM COS system is capable of providing standard vaults and retention enabled vaults. A vault is a logical storage partition within an IBM COS system, equivalent to a bucket in AWS. Retention enabled vaults enforce retention protection for all objects stored in that vault.
Objects can either be stored using the standard AWS S3 API or an S3-compatible API extension, where the latter is providing further retention control . When objects are stored with the standard S3 API a predefined retention periods is applied to each object at the time the object is stored. The predefined retention period is defined on vault level. When using the S3-compatible API retention extensions, the retention period can additionally be given with the API request. In addition, the S3-compatible API extension allows to extent the retention period for individual objects and set and remove legal holds.
Objects stored in a retention enabled vault cannot be deleted nor modified during the retention period. Versioning is disabled by default for retention enabled vaults. After the retention period has expired, objects can be deleted if there is no legal hold set on the object. Thus legal holds provide an additional level of protection for objects and can be used by applications to control the object lifecycle.
The IBM COS immutable storage function has been assessed for compliance by independent auditors according to German, Swiss and French laws  and in accordance to the US Security and Exchange Commission (SEC).
Configuring a vault with retention enabled
We assume the IBM COS system has been setup and initialized already. In order to create a retention enabled vault, the system vault protection setting in the system configuration menu of the IBM COS Manager web interface must be enabled. Select the tab Configure and select Configure Vault Protection:
To create a vault with retention enabled, launch the vault creation wizard and select Enabled at the Retention setting as shown below:
The Retention setting Enabled means that the vault is enabled for retention. Once set, this cannot be changed later. All objects stored in this vault will be protected during a retention period. The following retention period limits can be specified:
Managing vault access
Once a retention enabled vault has been created, users can be entitled to upload and download objects into the vault. This requires to create a user and allow access for the user to the vault. Select the Create Account wizard under the Security menu. Enter the user name, password, roles and vault access permissions, as shown below:
In the screen above select Move to Owner or Move to Read/Write in order to grant the user access to the vault named Rete
Once the user has been created and granted access to the vault, you have to obtain the user credentials. The user credentials for the S3 API are represented by an access key and secret key. These keys can be obtained from the Security menu, select the user and obtain the keys as shown below:
These user credentials now have to be used for any object storage operation.
Some basics about the API
Retention vaults can be used with the standard S3 API. Thus standard S3 PUT operations can be used to upload an object, GET operations can be used to download an object, HEAD operations can be used to show the object metadata and DELETE operations can be used to delete an object. Deletion however is only possible if the object retention period has expired and no legal hold is set. In addition, the standard S3 API does not allow to manage retention times. This means objects inherit the default retention period predefined for the vault. Find below some examples for the S3-compatible API extension that allows to manage object retention.
Important notice: Retention enabled vaults enforce the use of AWS signature version 4. In addition, when uploading objects, the content MD5 has to be provided with the request.
To manage object retention periods and legal holds, the S3-compatible API extension has to be used. For example, to PUT an object with a retention period, the following header has to be provided with the PUT request:
The tag Retention-Period specifies the retention period in seconds. Alternatively, the tag Rete
It is also possible to extend the retention period using the POST operation. Here is an example POST request:
The tag Addi
It is also possible to place a legal hold on the object. An object with at least one legal hold cannot be deleted even if the retention period has expired. A legal hold can be set with a PUT operation or after the object has been uploaded using a POST operation. An object can have more than one legal hold. This enables the implementation of different lifecycle management policies, where each policy uses its own set of legal hold IDs. To post a legal hold, the following operation can be used:
The tag legal-hold-ID is a user-defined name string given to the legal hold. It must be unique for an object.
Likewise a legal hold can be removed using the POST call:
It is also possible to view the vault protection setting, i.e. the predefined retention periods (default, minimum and maximum). Here is an example for the corresponding GET operation:
In the next section we provide some tooling based examples that illustrate how to use retention enabled vaults.
Examples for using retention enabled vault via API
After we have created a vault with retention enabled we can start using it. Objects can be uploaded, downloaded, listed and deleted using the standard S3 API or the S3-compatible API extensions. Retention enabled vaults enforce the use of Content-MD5 and AWS Signature version 4, regardless if the standard S3 API or the S3-compatible API extension are used.
Content-MD5 is a mechanism to verify the integrity of the data transferred via HTTP. Thereby the MD5 algorithm is applied on the raw data that is being transmitted and the resulting MD5 checksum is added to the header of the HTTP request by the sender. The receiver applies the MD5 algorithm on the raw data again and compares the resulting MD5 checksum with the checksum in the HTTP request.
With an AWS Signature HTTP requests sent to the Object Storage are signed, allowing the receiver (Object Storage) to determine the identity of the sender (Object Client). AWS Signature version 4 uses the AWS access key to compute the signature. The computed signature must be added to the request header.
In this chapter we provide examples for using retention enabled vaults with standard tools and some special code that incorporates the S3-compatible API extensions.
Using the standard S3 API
When using the standard S3 API, the normal object storage API calls such as PUT, GET, LIST and DELETE can be used. On standard PUT-requests the default retention period configured for the vault is applied for the object. Objects can only be deleted if the retention period has expired. This is checked by the IBM COS system and if the retention period has not expired the DELETE request receives an error 451 indicating “Unavailable for legal reasons”.
The standard S3 API does not allow to explicitly set a retention period for an object or to set or remove legal holds on objects. It does also not allow to show the retention period of an object and legal holds. In order to do this, the S3-compatible API extension is required.
To exploit the standard S3 API the AWS command line interface can be used . After installing and configuring the tool, it can be used to upload, download and list objects in a retention enabled vaults. However, it does not have a notion for managing retention periods. Find below some guidance for using this tools.
Once the tool has been installed, you have to configure the user credentials. This is done with the following command that will store the credentials in ~/.aws/credentials by default.
# aws configure
You can configure multiple users with different credentials. To configure different profiles with different user credentials use the command:
# aws configure --profile profile-name
In order to use a particular profile use the option --profile profile-name with the commands shown below.
List all vaults (buckets) configured in the system:
# aws [--profile profile-name] --endpoint-url http://cos1 s3 ls
List the content of a particular vault named “myvault”:
# aws [--profile profile-name] --endpoint-url http://cos1 s3 ls s3://myvault
Upload a file named “myobject” to the vault “myvault”
# aws [--profile profile-name] --endpoint-url http://cos1 s3 cp ./myobject s3://myvault
List an object:
# aws [--profile profile-name] --endpoint-url http://cos1 s3 ls s3:/
Download an object:
# aws [--profile profile-name] --endpoint-url http://cos1 s3 cp s3:/
The deletion of an object may fail if the object still has a retention period:
# aws [--profile profile-name] --endpoint-url http://cos1 s3 rm s3:/
If the retention period of the object has not expired, the delete operation will fail with the following error message:
delete failed: s3:/
As you can see, the standard API allows basic operations, but it does have any capability to manage retention periods and legal holds for objects. In order to find out more about the object retention, the S3-compatible API extension should be used.
Using the S3-compatible API extension
The S3-compatible API extensions allows to manage the retention for objects. In this section, we will give you some coding examples for managing object retention in a retention enabled vault. These examples are based on Python and require the modules requests and requests-aws4auth.
In order to initialize AWS version 4 authentication method, use the following method provided by the AWS4Auth module, whereby the parameters ACCESS_KEY and SECRET_KEY are the user credentials outlined in section Obtain the user credentials. The region denotes the region of the object storage:
The object auth is used within the next examples.
The following example shows how to PUT an object with an explicit retention period. The retention period is specified as number of seconds. The object with name myretobject will be uploaded to bucket mybucket to the accesser with the DNS alias cos1:
Instead of the retention period, the expiration date can be given. The expiration date must be given in ISO 8601 format, e.g. YYYYMMDDThhmmssZ. The same PUT method as shown above can be used, the only difference is the header tag:
Make sure that the object name does not already exist in the vault
The HEAD operation shows some interesting details about the object, including the expiration date and the number of legal holds. The requests class allows to decode the header content through the returned response object:
The next examples shows how to extend the retention period using the POST operation. The retention period is given in seconds, in this example we extend the retention by 86400 seconds which is one day:
Instead of specifying a retention period to be added to the current expiration date, a new expiration date can be set for the object. The new date must be greater than the old date. The difference to the previous example is the header tag:
It is also possible to set one or more legal holds on an object. The object cannot be deleted as long as a legal hold exists. The following example shows how to set a legal hold with the ID myhold1:
Likewise a legal hold can be removed, the difference to the above example is the request modifier as part of the URL:
As shown above, the S3-compatible API extension provides a comprehensive set of operations to manage retention on objects. It is also possible to show the retention setting such as object expiration time and legal holds using a GET request:
The object parameter response.text shows the XML encoded retention details of the object. Find an example below:
The object in this examplehas been created on 28 May 2018 19:32:19 and has a retention period of 520060 seconds which correlates to an expiration date of 03 Jun 2018 20:00:00. In addition this object has one legal hold named myhold1.
IBM COS retention enabled vaults allow storing objects in a writ
The addition of retention management functions to the standard S3 API is a bold step that the IBM COS team has taken. It enables application developers who are familiar with the widely accepted S3 API to quickly adopt IBM COS retention enabled vaults and manage archive data in accordance to legal demands. And it enables users and applications to truly store all kind of data in an IBM COS system, including data that must be protected in a WORM fashion.
IBM COS retention enabled vaults are one of the key storage offerings addressing regulatory compliance. For this purpose, the IBM COS team collaborates with key software vendors such as Veritas Enterprise Vault, IBM Filenet and NICE in order to enable applications using IBM COS retention enabled vaults.
 IBM Redpaper: “IBM Cloud Object Storage - Concepts and Architecture - An Under-the-Hood Guide for IBM Cloud Object Storage”
 IBM COS application programming interface reference:
 Amazon Web Services command line interface
 Assessment report of IBM COS regarding US SEC 17a-4f rules created by Cohasset Associates:
 IBM Redbook: “Cloud Object Storage as a Service: IBM Cloud Object Storage from Theory to Practice - For developers, IT architects and IT specialists”
 W3C date and time formats
 IBM COS Assessment by and independent auditor in accordance to German, Swiss and French laws:
Software attestation report: http
This document reflects the understanding of the authors in regard to questions asked about archiving solutions with IBM hardware and software. This document is presented “As-Is” and IBM does not assume responsibility for the statements expressed herein. It reflects the opinions of the author. These opinions are based on several years of joint work with the IBM Systems group. If you have questions about the contents of this document, please direct them to the Author (nil
The coding examples provided here in are for illustrational purposes only. There is no guarantee that these examples work. In addition IBM does not assume any liability caused by damages using these coding examples.
The following terms are trademarks or registered trademarks of the IBM Corporation in the United States or other countries or both: IBM logo, IBM Cloud Object Storage.
Amazon Web Services (AWS) is a [registered} trademark of Amazon Inc. in the United States and other countries.
Other company, product, and service names may be trademarks or service marks of others.