Configure an pipeline to load files from a Amazon S3 bucket, Amazon SFTP folder, or a
mailbox. If you have already transformed the data into the correct format, you can import the data
without transforming it.
About this task
The pipeline loads files based on a filename pattern that you specify when you configure the
pipeline.
Note: The filename pattern must be configured to limit the possibility of a file matching
multiple pipelines. If a file matches multiple pipelines, it causes an error and the file is not
processed.
When you associate the data pipeline with a data service, select all applicable data services,
unless there is a specific reason to limit the source to specific data services.
Note: You cannot
delete a pipeline if it was used to download or transform a file or if it is connected to a data
source. Disable the pipeline first.
Procedure
-
As a system administrator, from the main menu, click
to open the Data Services grid.
- On the Data Pipeline tab, click Add New
Pipeline.
- Assign a name to the pipeline.
- Add a description.
- Set the type of pipeline to a file loading.
- Set the target system to the accounts or meters, based on whether the files contain
account data or meter readings. You can also choose attachments if the pipeline is being used for
the bulk attachment upload process.
- Enter a unique filename pattern by using regex. The data pipeline receives the files that
matches the filename pattern.
- In the data source field, select the associate data services that will receive the
files.
- Leave Data Transformer as None
(pass-through).
- Optional: For account and meter data files that do not require
transformation, you can leave the Additional Parameters field blank. If you
are using the pipeline to load attachments, you can choose to add the following additional
parameters:
- "FileProcessingDelayInMinutes":<value>
- This parameter is useful when account records and attachments are loaded at the same time, to
ensure that the account record is loaded before the attachment.
- "FileNameFormat":"[AssociateName]~Item~[FileName][FileExtension]"
- This parameter is useful where the original filename of the attachment is the account record
reference. You can use this parameter to automatically rename the attachment to match the
format that is required by the bulk attachment upload process.
Note: The additional parameters must be entered as a valid JSON string enclosed in curly brackets.
Standard quotation marks rather than enclosed quotation marks must be used.
For
example:
{"FileNameFormat":"[AssociateName]~Item~[FileName][FileExtension]", "FileProcessingDelayInMinutes":60}
- Click Save.
Results
The pipeline is created and is ready to process files.