Configuring a file transformation and loading pipeline

Configure a pipeline to load files from an Amazon S3 bucket, Amazon SFTP folder, or a mailbox. Select the IBM® App Connect Data Transformer to transform the file into the correct format for loading.

1. As a system administrator, from the main menu, click Admin > Data Flow Automation to open the Data Services grid.
2. On the Data Pipeline tab, click Add New Pipeline.
3. Assign a name to the pipeline.
4. Add a description.
5. Set the type of pipeline to a file loading.
6. Set the target system to the accounts or meters, based on where the files contain account data or meter readings.
7. Enter a unique filename pattern by using regex. The data pipeline receives the files that matches the filename pattern.
8. In data source, select the associate data services that will receive the files.
9. In Data Transformer, select the App Connect ETL data service.
10. Optional: Set additional parameters to specify how App Connect will parse the files. Configure how the files are parsed by App Connect by setting these parameters. If the parameters are not entered, the following default values are used:
    Note: The additional parameters must be entered as a valid JSON string enclosed in curly brackets. Standard quotation marks rather than enclosed quotation marks must be used.

    For example:

    {
      "csvDelimiter": ".",
      "decimalSeparator": ",",
      "thousandsSeparator": ",",
      "csvEncoding": "UTF8"
    }

11. Click Save.