Creating an S3 storage device

S3 storage devices can be configured with an Advanced Storage Area.

Before you begin

Before you create an S3 device, you must do the following tasks:
  • Decide how to provide S3 credentials to the Content Platform Engine.

    There are two options: static credentials with a secret key, or using a credentials manager. With static credentials you set the credentials manually, and with a credentials manager they are provided on the fly by a credentials manager that retrieves them from a source you define. Credentials managers are useful if you need to periodically change credentials. For more information about credential managers see Credentials Managers for Amazon S3 storage devices.

  • If you choose to use static credentials for authentication, obtain credentials for connecting to the S3 object storage provider.

    To create an S3 storage device for IBM Cloud Object Store, you need to first create the HMAC credentials in IBM Cloud console. HMAC credentials consist of an Access Key and Secret Key paired for use with S3-compatible tools and libraries that require authentication. See the following IBM Cloud topic for more details: Using HMAC credentials External link opens a new window or tab

    When you view the user credentials, the following section represents the HMAC credential and includes the S3 access key ID and S3 secret access key you need to create the S3 advanced storage device:
    cos_hmac_keys:{
                 access_key_id: 7exampledonotusea6440da12685eee02
                 secret_access_key: 8not8ed850cddbece407exampledonotuse43r2d2586 
    }
  • If you choose to use a credentials manager for authentication, create the credentials manager that will provide the credentials to the S3 storage device. See Setting up a credentials manager with an EC2 role and Setting up a credentials manager with a web identity provider
  • Determine the device connection URL to the S3 storage.
  • Create an S3 bucket where the Content Cortex content will be stored.
    Note: You are responsible for configuring the S3 bucket and its related attributes outside of Content Platform Engine. You must pursue any issues that are related to configuration and set-up with the storage vendor.
  • If you want to specify a S3 storage class for the S3 storage device, you can use the JVM configuration parameter Content.S3.StorageClass.Value to specify the storage class. For example, you can set Content.S3.StorageClass.Value_{E8CCFED2-3B14-CFD4-8546-8E4925300000}=GLACIER_IR, where, the GUID is the object ID of the S3 storage device. When you configure the JVM parameter for the S3 storage device, all content that is uploaded to the S3 storage device is stored by using the GLACIER_IR storage class.
  • The Content Platform Engine supports the use of the AWS S3 Intelligent-Tiering storage class for both S3 advanced storage devices and S3 fixed content devices. The S3 Intelligent-Tiering storage class optimizes storage costs by automatically moving data between access tiers based on changing access patterns.

    The Content Platform Engine supports only Frequent Access, Infrequent Access, and Archive Instant Access tiers. Do not activate the Archive Access and Deep Archive Access tiers in AWS when you use S3 Intelligent-Tiering storage class with Content Platform Engine.

    To use the S3 Intelligent-Tiering storage class with Content Platform Engine, you must set the JVM parameter as -DContent.S3.StorageClass.Value_{Device_ID}=INTELLIGENT_TIERING where, {Device_ID} is the object ID of the S3 advanced storage device or S3 fixed content device.

    Alternatively, you can achieve the same goal by creating a lifecycle management policy in AWS to move content between storage classes, allowing for more customized and flexible configuration options.

  • Determine whether an SSL connection will be used between the Content Platform Engine and the S3 storage.
For IBM Cloud Object Storage, only TLS v1.2 HTTPS protocols are supported. To ensure that the correct HTTPS protocols are used, add the following JVM argument:
-Dhttps.protocols=TLSv1.2
How you configure the certificate might be different depending on the application server that you're using:
  • If your Content Platform Engine is running on WebSphere® Application Server, you must deploy the S3 client certificate directly in the WebSphere Application Server Administrative Console. For details, see Deploying a client certificate on WebSphere.
  • If your Content Platform Engine is running on Oracle WebLogic Server, you can export the certificate from the region's host, for example, https://s3-us-west-1.amazonaws.com/, and then import the WebLogic JRE as a signer certificate.
Tip: The Content Platform Engine S3 Advanced Storage connector does not use the S3 versioning feature. It is recommended that versioning not be configured on the S3 bucket used by the Content Platform Engine. However, the Content Platform Engine can tolerate any S3 bucket version setting, but be aware that using version enabled or version suspended settings on your S3 bucket can complicate content deletion. In these cases, to make sure content is really deleted from your S3 bucket, set the Advanced.S3.DeleteSpecificVersion property to True, either in the FileNet.properties file or as a JVM argument. To delete a specific version of a document you need the additional S3 permissions: s3:DeleteObjectVersion DELETE Object

Procedure

To create an S3 storage device:

  1. Start the New S3 Device wizard in the administration console:
    1. In the tree view, click the Object Store > object store name to open the object store that uses the device.
    2. In the object store tree view, right-click the Administrative > Storage > Advanced Storage > Advanced Storage Devices folder and click New S3 Device.
  2. Complete the wizard.
    The values that you enter into the wizard fields can differ depending on what kind of S3 storage device you are creating and how you choose to provide credentials.
    You set the general values for the storage device, then in a second window, if you set Authentication Method to Secret Key you enter the S3 Access Key ID and S3 Secret Key, or, if you set Authentication Method to Credentials Manager, you set the credentials manager you want to use for the storage device.
    Table 1. S3 Storage Device Wizard values
    Field Value
    Device connection URL The URL value defines the endpoint for the storage device to access the S3 bucket. The value can be a path-style access URL, which must include the bucket name, or a hosted-style access URL.
    For an S3 device, the URL value might look like one of the following examples:
    Path-style access
    https://s3.us-west-1.amazonaws.com/mybucket
    Virtual hosted-style access
    https://mybucket.s3.us-west-1.amazonaws.com/mybucket

    Amazon recommends using virtual hosted-style access. For information about support plans, see Amazon S3 Path Deprecation Plan External link opens a new window or tab.

    For an IBM Cloud Object Storage device, the URL value might look like one of the following examples:
    Path-style access
    https://s3.us-east.cloud-object-storage.appdomain.cloud/mybucket
    Virtual hosted-style access
    https://mybucket.s3.us-east.cloud-object-storage.appdomain.cloud

    The endpoint, s3.us-east.cloud-object-storage.appdomain.cloud in this example, can be found in the IBM Cloud console by checking the Endpoints section under the Bucket Configuration.

    S3 Access Key Id Enter the AWS access key ID for the AWS account or for an AWS Identity and Access Management (IAM) created user.
    S3 Secret Key Enter the S3 secret access key.
    S3 bucket name Note that the S3 storage device implementation does not create the bucket automatically if the specified bucket does not exist. Without a value for an existing bucket, the wizard will not complete.
    S3 region name When you specify an S3 Region Name for an Amazon S3 storage device, use a value from the Region column in the following table instead of a value from the Region Name column:

    http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region External link opens a new window or tab

    For IBM Cloud Object Storage devices, set the S3 region name value to us-standard.

    HTTPS certificate validation All S3 regions support both HTTP and HTTPS connections. To use an HTTPS connection, a valid SSL certificate must be installed on each Content Platform Engine server.