Using Google Cloud Storage as a S3 Advanced Storage Device in Content Platform Engine

Starting with Content Platform Engine  5.5.0, you can create a Simple Storage Service (S3) Advanced Storage Device in Content Platform Engine.  A Content Platform Engine S3 Advanced Storage Device is implemented with the Amazon published S3 REST API and works with different AWS S3 compatible storage devices including Amazon S3, IBM Cloud Object Storage (ICOS) and Dell/EMC ECS Object Storage.

Google Object Storage provides a similar interface to its storage as Amazon S3 does. It has defined its own full set of REST-based XML APIs and JSON APIs, but these are not compatible with Amazon S3 APIs.  However, Google Object Storage does provide AWS S3 access through an S3-compatible API.  You can use Google Cloud Storage as a Content Platform Engine S3 Advanced Storage Device with certain restrictions and limitations.

The Google Object Storage documentation (https://cloud.google.com/storage/docs/migrating), describes two different ways for using the AWS S3 interface to access Google Cloud Storage. To support Google Object Storage as a Content Platform Engine S3 Advanced Storage Device, you must follow the instructions describing the Simple Migration approach to setting up your project and bucket and obtaining the Google Cloud Storage Hash-base Message Authentication Code (HMAC) access key and secret for your credentials.

Here is an example of the configuration values for setting up a Google Object Storage as a Content Platform Engine S3 Advanced Storage Device:

Device Connection URL:       https://storage.googleapis.com/my-test-bucket

Device User Name:                GOOG6ATSRSR6O8O7TSSGFRD2    (HMAC access key)

Device User Password:           DGFQYTHb8b9HVu9ufjc9JRcgPfLFTN6g8J4BLeSa(HMAC secret)

S3 Bucket Name:                    my-test-bucket

S3 Region Name:                    europe-west2                       

The following restrictions apply when using Google Object Storage as a Content Platform Engine S3 Advanced Storage Device:

• You must follow the Google Object Storage documentation for the “Simple Migration” approach to set a default Google Project and obtain HMAC keys.

• The Google Object Storage Bucket needs to be version disabled.

• Google Object Storage now support S3 multi-part content uploads. But the implementation is not fully compatible with AWS S3. The initiating multi-part upload is a simple POST call without a request body and should NOT require a content-length header according to AWS APIs. IBM did implement a work-around for this in 5.5.9 for those S3 solutions that mistakenly require content-length header, but IBM will not port the workaround back to any earlier versions.

  • For customers on CPE 5.5.9 and later release, to use multi-part content upload with Google Cloud Storage, the customer can set Content.S3.InitMultiUpload.UseEmptyRequestBody to true using the FileNet.properties file or as a JVM parameter. For information on how to set this configuration parameter, please refer to the technote (https://www-01.ibm.com/support/docview.wss?uid=swg21496579).

  • For customers on CPE versions prior to 5.5.9, you must continue to only use single part upload for Google Cloud Storage. The Content Platform Engine S3 implementation uses S3 multi-part uploads with a default part size of 5MB. This means that for any content larger than 5MB in size, the upload to Google Object Storage will fail.  In order to handle large content uploads, you need to set the configuration parameter Advanced.S3.StreamUploadPartSize using the FileNet.properties file or as a JVM parameter. For information on how to set this configuration parameter, please refer to the technote (https://www-01.ibm.com/support/docview.wss?uid=swg21496579).

    For example, by setting Advanced.S3.StreamUploadPartSize=20971520 (which is 20MB), you can upload content up to 20MB in size to Google Object Storage.  Please note that by default, Content Platform Engine will hold the content of an upload part in a memory buffer. When setting the configuration parameter value, you must consider the memory resources available on your Content Platform Engine system.  For example, if you have 5 clients doing concurrent uploads with a Part Size set to 20MB, your system will need at least 100MB memory for these operations.

    If you need to upload very large content and don’t have enough memory resources to support these uploads, you can choose to use the Content Platform Engine S3 staging file upload approach which won’t load the content into a memory buffer and can handle much larger content uploads. To do that, you need to set the following two configuration parameters, for example:

    Advanced.S3.UseFileStagingUpload=true

    Advanced.S3.FileUploadPartSize=209715200  (200MB, the default value is 10MB)

    This will first stream the content into a temporary file in the S3_Staging folder under your Content Platform Engine server domain context temporary folder, for example for WebSphere Server, it will be a folder like:

    C:\IBM\WebSphere\AppServer\profiles\<profile_name>\FileNet\<server_instance_name>\S3_Staging

• The Content Platform Engine S3 Advanced Storage Device does not set object ACLs and the Storage Class during the upload.  For Google Object Storage, this means Default “project-private” ACL (Objects are always owned by the original requester who uploaded the object) and the bucket default Storage Class will be applied to the uploaded content.