Using the IBM Sterling Intelligent Promising file transfer adapter for data ingestion

By using the file transfer adapter, you can transfer various data types from other systems to your IBM Cloud® Object Storage bucket. These data types are important to the process of combining and harmonizing data from multiple sources into a unified, coherent format that can be used for various analytical, operational, and decision-making purposes.

Before you begin

Before you transfer data by using the file transfer adapter, complete the following step:

Download, install, and configure the file transfer adapter. For more information, see Downloading, Installing, and configuring the IBM® Sterling Intelligent Promising file transfer adapter.
Configure the IBM Cloud Object Storage bucket and endpoint for each data feed type. For more information, see Configuring endpoints for file ingestion rules.

About this task

The file transfer adapter transfers data sets from on-premises systems to your IBM Cloud Object Storage bucket to make it available for Sterling Intelligent Promising. You can upload the various data feed types that are supported. When the data is uploaded, it is analyzed for consistency and is used by Sterling Intelligent Promising for predictive optimization models.

Procedure

For each data feed that you want to transfer by using the file transfer adapter, complete one of the following steps:
- To run the file transfer adapter directly from the <install-dir> directory, enter:
```
cd <install-dir>
./sip_cos_file_adapter.sh <options>
```
- To run the file transfer adapter directly from any other directory, enter:
```
./path/to/<install-dir>/sip_cos_file_adapter.sh <options>
```
  The <install-dir> is the directory where you extracted the contents of the .zip file provided.
When you upload files to your IBM Cloud Object Storage bucket, by default, the files are stored in the <Input_Prefix> /UPLOAD/<dataType> virtual folder. The <Input_Prefix> and <dataType> are the values that are provided for the <Input_Prefix> and dataType configuration options. To store files in a different folder, use the path configuration option to specify the full virtual folder path within the bucket to upload the files. For more information, see the Sample commands that are provided.
By default, you can upload files only with the following content types: text/csv, text/plain, or application/gzip files. To upload files with other content types, add them to the comma-delimited values list of the supportedContentTypes option. For more information, see Downloading, installing, and configuring the IBM Sterling Intelligent Promising file transfer adapter.
Ensure that the files are uploaded successfully to IBM Cloud Object Storage. For more information, see Monitoring data files for data ingestion .
To download files from or to list files in your IBM Cloud Object Storage bucket, provide the path configuration option to specify the virtual folder path. A blank value for the path configuration option represents the root virtual folder.

Sample commands

Review the following sample commands that describe how to list supported configuration options and also complete file upload, download, and list operations:
List all configuration options
For example, to list all configuration options that are supported by using the command-line with a brief description, enter:
```
./sip_cos_file_adapter.sh -help
```
Upload a single file
For example, to upload a single file to the <Input_Prefix>/UPLOAD/INV/ virtual folder path, enter:
```
./sip_cos_file_adapter.sh -conf transfer.options -op UPLOAD -dataType INV -src /path/to/local/data_file.csv
```
Upload all files
For example, to upload all files from a local directory to the <Input_Prefix>/UPLOAD/INV/ virtual folder path, enter:
```
./sip_cos_file_adapter.sh -conf transfer.options -op UPLOAD -dataType INV -src /path/to/local/directory
```
Download all files
For example, to download all files from the <Input_Prefix>/UPLOAD/INV/ virtual folder path, enter:
```
./sip_cos_file_adapter.sh -conf transfer.options -op DOWNLOAD -path <Input_Prefix>/UPLOAD/INV -downloadDest /path/to/local/download/directory
```
For example, to download all files from your IBM Cloud Object Storage bucket, do not enter any value for the path option, enter:
```
 ./sip_cos_file_adapter.sh -conf transfer.options -op DOWNLOAD -path "" -downloadDest /path/to/local/download/directory
```
List all files
For example, to list all files, up to 10,000, in the <Input_Prefix>/UPLOAD/INV virtual folder path, enter:
```
./sip_cos_file_adapter.sh -conf transfer.options -op LIST -path <Input_Prefix>/UPLOAD/INV
```
For example, to list all files, up to 10,000, in the <Input_Prefix>/UPLOAD/INV virtual folder path, enter:
```
./sip_cos_file_adapter.sh -conf transfer.options -op LIST -path <Input_Prefix>/UPLOAD/INV
```
Understand the protocol that is generated for the prefix, for example:
- If you define your bucket prefix as /myInput, the system generates the following prefix automatically:
  - /failed/sip-catalog/myinput
  - /completed/sip-catalog/myinput
  - /reprocess/sip-catalog/myinput
  - /summary/sip-catalog/myinput
  Depending on the rules that are configured, you might see sip-data-capture included instead of sip-catalog.
Familiarize yourself with the file format, for example:
- If the original file is called 2024-01-01-10-59-00.0000-3423-Availability.csv, for simplicity, you might call this file <myfilename>.csv.
- If the file fails, the following format is shown:
```
<myfilename>_file_failed_row.csv. for the error rows only (all successful are not present here)
<myfilename>_file_failure_reason.txt are the summary file which provide the error cod
```