Using IBM Cloud Object Storage S3 with IBM App Connect Enterprise

IBM® Cloud Object Storage S3 is ideal for holding large amounts of colder production data, such as backups and archives, and very large individual files, such as video files, image files, and genomic data. IBM Cloud Object Storage S3 is a reliable, durable, and resilient object storage.

1. In the IBM App Connect Enterprise Toolkit, create a flow containing a IBM Cloud Object Storage S3 Request node.
2. Select the IBM Cloud Object Storage S3 Request node in the flow to show the node properties in the editor.
3. On the Basic tab, click Launch Connector Discovery.
   A panel is displayed in which you specify the name of the policy project and vault details to be used during connector discovery.
4. Specify the details of the policy project and vault to be used during connector discovery:
   1. In the Policy Project field, specify the policy project that is used to store the policies that are created during connector discovery.
      Alternatively, you can create a new policy project by clicking New and then specifying the name of the new policy project. Then click Finish.
   2. Specify the vault to be used during connector discovery. By default, credentials that are used during connector discovery are stored in an external directory vault, which is an App Connect Enterprise vault that can be used by any integration server. Alternatively, you can store the credentials in an integration server vault, which is created in the integration server's work directory and can be used only by that specific integration server.
      To specify the vault to be used for storing the credentials, complete the steps in the Using the Connector Discovery wizard section of one of the following topics:

      • Configuring a shared vault
      • Configuring an integration server vault
   3. In the Vault key field, enter the vault key that is used to access the credentials stored in the vault. The vault key must be at least 8 characters in length.
   4. Optional: By default, the specified vault location and vault key are saved as preferences in the Toolkit so that the values are preset when you launch Connector Discovery. If you do not want the preferences to be saved, deselect Save in vault preferences.
5. Click Launch Discovery to start the Connector Discovery wizard for the IBM Cloud Object Storage S3 connector.
   The Connector Discovery window is displayed. If existing IBM Cloud Object Storage S3 connections (accounts) are available, a list of those connections is displayed. If there are no existing connections, the status of the IBM Cloud Object Storage S3 connector is shown as Not connected.
   • If one or more IBM Cloud Object Storage S3 connections (accounts) are available, complete the following steps:
     1. Select the connection (account) that you want to use by clicking it.
     2. Click the required object type and then select the action that you want to perform on the object. For example, to retrieve buckets from IBM Cloud Object Storage S3, click Buckets and then Retrieve buckets.
   • If there are no existing connections (accounts), complete the following steps:
     1. Click the required object type and then select the action that you want to perform on that object. For example, to retrieve buckets from IBM Cloud Object Storage S3, click Buckets and then Retrieve buckets.
     2. Click Connect to display a menu from which you must select one of the following authorization methods:
        • Provide credentials for App Connect to use (BASIC IAM)
        • Provide credentials for App Connect to use (BASIC)
     3. If you selected Provide credentials for App Connect to use (BASIC IAM) as the authorization method, enter the following details:
        1. Endpoint URL:Specify the URL of the Cloud Object Storage Endpoint. Only public endpoints are supported.
        2. API key: Specify the API key of the instance if the service instance uses the Identity and Access Management (IAM) authentication.
        3. Resource instance ID: Specify the resource instance ID of the instance if the service instance uses the Identity and Access Management (IAM) authentication.
        4. Application client secret: Enter the application client secret for a project-specific unique application client ID.
     4. If you selected Provide credentials for App Connect to use (BASIC) as the authorization method, enter the following details:
        1. Endpoint URL Specify the URL of the Cloud Object Storage Endpoint. Only public endpoints are supported.
        2. Secret access key (optional): Specify the secret access of the instance if the service instance uses HMAC credentials for authentication.
        3. Access key ID (optional): Specify the access key ID of the instance if the service instance uses HMAC credentials for authentication.
        4. Region (optional): Specify the secret access of the instance if the service instance uses HMAC credentials for authentication.
        5. API key (optional): Specify the API key of the instance if the service instance uses the Identity and Access Management (IAM) authentication.
        6. Resource instance ID (optional): Specify the resource instance ID of the instance if the service instance uses the Identity and Access Management (IAM) authentication.

        For more information about accessing or generating these connection details, see How to use IBM App Connect with IBM Cloud Object Storage S3 in the IBM App Connect Enterprise as a Service documentation.

     5. Click Connect.
6. Set the required connector properties in the wizard.
   For retrieve or update actions, you can add conditions for the retrieval of the data by clicking Add condition and then selecting the property that you want to filter on.

   If you add conditions for retrieve or update actions, you can optionally use condition filtering to refine the conditions that are applied. To use condition filtering, exit the Connector Discovery wizard by clicking the Close button (X) and then complete the instructions in Using condition filtering.

   For create actions, you can optionally use advanced mode. In the default edit view for an action, some applications have fields that are hidden because they are not required for general use cases. For more advanced use cases, you can switch to advanced mode editing, which provides extra capabilities for editing flows. To use advanced mode, exit the Connector Discovery wizard by clicking the Close button (X) and then complete the instructions in Using advanced mode.

   You can also set properties that specify the maximum number of records to retrieve and the action to be taken if that limit is exceeded.

7. When you have finished specifying the properties in the Connector Discovery wizard, click Save.
   The credentials that are used for connecting to IBM Cloud Object Storage S3 are stored in the vault, and the other connection details are saved in the IBM Cloud Object Storage S3 policy. For more information, see IBM Cloud Object Storage S3 policy. The values of the properties that you set in the wizard are returned to the IBM Cloud Object Storage S3 Request node in the IBM App Connect Enterprise Toolkit.
8. When you have finished discovery and saved the property values, exit the Connector Discovery wizard by clicking the X in the upper-right corner of the window or by pressing Alt+F4.
9. Return to editing the IBM Cloud Object Storage S3 Request node in the IBM App Connect Enterprise Toolkit.
   The connector properties that were set in the Connector Discovery wizard (in step 6) are now visible on the IBM Cloud Object Storage S3 Request node in the property editor. The Basic tab shows the values of the Action and Object properties that you set in the wizard. For example, if you selected Buckets > Retrieve buckets in the wizard, the following properties will be visible on the Basic tab of the node:

   • Action - RETRIEVEALL
   • Object - bucket

   The values of the Action and Object properties are displayed in read-only format. If you want to change these values, you can do so by clicking Launch Connector Discovery again and setting new values in the Connector Discovery wizard.

   The Schema base name property specifies the base name of the schema files that describe the format of the request and response messages that are sent to and received from the IBM Cloud Object Storage S3 connector. The schema base name is set automatically the first time that you run discovery for the node, and it is based on the current flow name and node name. If you set this property manually before running discovery for the first time, the value that you set will be used. If you rename the schemas after discovery, you must edit this property so that it matches the schema base name that is used by the renamed schemas in the project. If you change this property after discovery, you must either rename the schema names to match or run discovery again.

   Depending on the action that was selected during discovery, the Connector Discovery wizard generates either a request schema and a response schema, or a response schema only. A request schema is generated only if the selected action and object require a request message. The generated request schema is used for validation of the request message. If the action was RETRIEVE or DELETE, only the response schema is returned by the connector.

   The generated schema files are added to the project and can be used by a Mapping node for transforming input or output data. The full filename of the schema is derived from the schema base name (such as gen/MyMessageFlow.IBM_Cloud_Object_Storage_S3_Request), suffixed with either response.schema.json or request.schema.json. You can open the schema by clicking Open request schema or Open response schema.

10. Check that the property settings on the IBM Cloud Object Storage S3 Request node are correct and then save the message flow.
11. On the Connection tab of the IBM Cloud Object Storage S3 Request node, the Policy property shows the name of the policy that contains the details of the security identity to be used for the connection. The policy has a type of IBM Cloud Object Storage S3.
    For more information, see IBM Cloud Object Storage S3 policy.
12. Optional: Set the Timeout property on the Connection tab to specify the time (in seconds) that the node waits for IBM Cloud Object Storage S3 to process the operation.
13. The Filter tab of the IBM Cloud Object Storage S3 Request node contains properties that control the way in which the message flow selects data. The initial values of these properties are taken from the property values that were set for the IBM Cloud Object Storage S3 connector in the Connector Discovery wizard, including the filter options properties and any conditions that were specified (as described in step 6). If you subsequently return to the Connector Discovery wizard and change the values of any properties (by adding new conditions, for example) those updates are reflected in the properties set on the node.

    The Filter Options properties control which objects are to be operated upon when the IBM Cloud Object Storage S3 Request node executes. The Filter Limit properties control the maximum number of items to be retrieved and the action to be taken if the limit is exceeded.

    You can modify the values by clicking Edit next to the value that you want to modify in the Filter Options section, and by changing the property values that have been set in the Filter Limit section.

    The property values can be either text values or ESQL or XPATH expressions that are resolved from the contents of the message that is passed to the IBM Cloud Object Storage S3 Request node as it executes.

14. On the Request tab, set the Data location property to specify the location in the incoming message tree that contains the object data to be created in IBM Cloud Object Storage S3. This data forms the request that is sent from the IBM Cloud Object Storage S3 Request node to IBM Cloud Object Storage S3.
15. On the Result tab, set the Output data location property to specify the location in the output message tree that will contain the data of the record that is created in IBM Cloud Object Storage S3.
16. By default, request messages are validated against the request schema that was generated during connector discovery. You can turn off request validation or change the validation settings by using the Validation properties of the IBM Cloud Object Storage S3 Request node.
17. Save the message flow.