Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

watsonx.data on IBM Software Hub

watsonx.data Developer edition

If you select Amazon S3 from the Storage section, configure the following details:

Field Description
Display name Enter the name to be displayed.
Bucket name Enter the name of your existing bucket.
Region Select the region where the storage is available.
Endpoint Enter the Endpoint URL. For example, https://s3.us-west-2.amazonaws.com .
S3 access point topology Select the type of S3 endpoint to use:
  • Regional endpoint - Use a standard regional S3 endpoint for your bucket.
  • Global access point (MRAP) - Use an Amazon S3 Multi-Region Access Point (MRAP) for global access to your data across multiple AWS regions.
MRAP ARN Enter the Amazon Resource Name (ARN) of your Multi-Region Access Point. The format of the value is arn:aws:S3::<AWS_ACCOUNT_ID>:accesspoint/MultiRegionAccessPoint_<alias>
Note: This field appears only when Global access point (MRAP) is selected.
Region Select the region where the storage is available.
Endpoint Enter the endpoint URL.

For Regional endpoint, the format of the value is https://S3.<region>.amazonaws.com.

For Global access point (MRAP), the format of the value is https://<mrap-alias>.accesspoint.s3-global.amazonaws.com.

Authentication Mode You can select one of the following authentication methods:
  • HMAC credentials: Select this method to use access key and secret key as the authentication mechanism.
  • IAM Role: Select this method to use AWS Security Token Service (STS) that offer enhanced security compared to access and secret keys.
Role ARN The Amazon Resource Names (ARNs) uniquely identify AWS resources based on IAM role, which is an AWS identity with specific permission. Enter the value in the format arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>.
Note: This field is available only when the Authentication Mode is set to IAM Role.

To generate Role ARN, see Configuration required to retrieve Role ARN.
Access key Enter your Access key.
Secret key Enter your Secret key.
Input method
Note: This field is available only when use of secrets from an external vault (HashiCorp) is configured for your connections and also if you selected Authentication Mode as HMAC credentials.
 Choose the credential input method:
  • Enter credential manually
  • Use secret from vault
To use secrets from an external vault, select the Use secret from vault option and complete the following steps:
  1. Click Select secret.
  2. Select one or more secrets and click Select.

Access key

Secret key

If you are using secrets from vault, then select the Access key and Secret key from the respective drop-down lists. Otherwise, enter your Access key and Secret key in the respective fields.
Connection status Click the Test connection link to test the bucket connection. If the bucket connection is successful, a success message appears.
Designate this bucket as the ACL store Use the toggle switch to designate this bucket as the ACL store. If you enable the toggle switch, an Enable Access Control List (ACL)? dialog appears, click Enable.
Note: This feature applies to watsonx.data Premium. For more information on how to use this feature, see Governance through Access Controlled Lists (ACL).

If you enable the toggle switch, the Associate catalog option is selected by default, with the Apache Iceberg catalog preselected. You cannot choose a different catalog for ACLs. You can designate only one storage as the ACL store per instance. After a storage is designated, this option will no longer be visible or available.
Associate catalog Select the checkbox to add a catalog for your storage. This catalog is associated with your storage and serves as your query interface with the data stored within.
Catalog type Select the catalog type from the list. The recommended catalog is Apache Iceberg. The other options for catalog are Apache Hive, Apache Hudi, and Delta Lake.
Catalog name Enter the name of the associated catalog.
Base path (optional)
Note: This field appears only when you select Apache Iceberg as the catalog type.
 Enter the base path for the catalog in the object storage. This allows you to associate multiple Iceberg catalogs with a single storage.
Note: You cannot share a storage between Iceberg and non-Iceberg catalogs.
Associate Click Associate to create the storage.