Data loading using custom API connector

Edit online

Retrieve account or meter data from a client or supplier API and transform the data for loading into IBM® ESG Suite.

A Connector AddOn is required for data transformation and a Custom API Integration is required through an IBM Expert Labs Scope of Work. For more information, see the Connectors section.

To retrieve account or meter data from a client or supplier API and transform the data in App Connect, create both a file loading data service and an App Connect transformation data service, and associate both with a file loading data pipeline.

The data retrieved from a client or supplier API will be consolidated into raw data files and loaded to the file loading data service. The raw data files will then be transformed in App Connect like any other custom connector.

The end-to-end flow is as follows:
  1. Create a data service under the category File Loading.
    • To load data from an Amazon S3 bucket, set the data service type to Amazon S3 (Cloud Storage). For more information, refer to Amazon S3 data service.
    • To load data from Amazon SFTP, set the data service type to Amazon Transfer (SFTP). For more information, refer to Amazon SFTP data service.
    • To load data from an Amazon mailbox, set the data service type to Amazon Simple Email (Mailbox). For more information, refer to Amazon mailbox service.
  2. Create an App Connect data service under the Transformation category. For more information, refer to Provisioning IBM App Connect.
  3. Configure a file loading pipeline.
    1. Specify accounts or meters as the target system.
    2. Specify a filename pattern to use when looking for files to load.
    3. Select the data services to poll.
    4. Select the App Connect Data ETL service as the data transformer.
    5. Specify any additional parameters required by App Connect.
    For more information, refer to Configuring a file transformation and loading pipeline.