Configuring a file loading pipeline (without transformation)

Edit online

Configure a pipeline to load files from a Amazon S3 bucket, Amazon SFTP folder, or a mailbox. If you have already transformed the data into the correct format, you can import the data without transforming it.

About this task

Custom pipelines are required to load and transform account or meter data files in the following scenarios, where the incoming filename pattern does not match a system pipeline filename pattern:
  • To load files that are formatted in a client or supplier specific format through a legacy connector
  • To load files through new connectors that go through App Connect
  • To load and transform account or meter data files from a client or supplier API by using a custom API connector
  • To load files through bulk upload templates that require special handling, such as where additional parameters are used to rename or delay a file, for example, for loading attachments
The pipeline loads files based on a filename pattern that you specify when you configure the pipeline.
Note: The filename pattern must be configured to limit the possibility of a file matching multiple pipelines. If a file matches multiple pipelines, it causes an error and the file is not processed.
When you associate the data pipeline with a data service, select all applicable data services, unless there is a specific reason to limit the source to specific data services.
Note: You cannot delete a pipeline if it was used to download or transform a file or if it is connected to a data source. Disable the pipeline first.

Procedure

  1. As a system administrator, from the main menu, click Admin > Data Flow Automation to open the Data Services grid.
  2. On the Data Pipeline tab, click Add New Pipeline.
  3. Assign a name to the pipeline.
  4. Add a description.
  5. Set the type of pipeline to a file loading.
  6. Set the target system to the accounts or meters, based on whether the files contain account data or meter readings. You can also choose attachments if the pipeline is being used for the bulk attachment upload process.
  7. Enter a unique filename pattern by using regex. The data pipeline receives the files that matches the filename pattern.
  8. In the data source field, select the associate data services that will receive the files.
  9. Leave Data Transformer as None (pass-through).
  10. Optional: For account and meter data files that do not require transformation, you can leave the Additional Parameters field blank. If you are using the pipeline to load attachments, you can choose to add the following additional parameters:
    "FileProcessingDelayInMinutes":<value>
    This parameter is useful when account records and attachments are loaded at the same time, to ensure that the account record is loaded before the attachment.
    "FileNameFormat":"[AssociateName]~Item~[FileName][FileExtension]"
    This parameter is useful where the original filename of the attachment is the account record reference. You can use this parameter to automatically rename the attachment to match the format that is required by the bulk attachment upload process.
    Note: The additional parameters must be entered as a valid JSON string enclosed in curly brackets. Standard quotation marks rather than enclosed quotation marks must be used.
    For example:
    {"FileNameFormat":"[AssociateName]~Item~[FileName][FileExtension]", "FileProcessingDelayInMinutes":60}
  11. Click Save.

Results

The pipeline is created and is ready to process files.