Monitoring data pipelines

Edit online

You can monitor data pipelines and troubleshoot data flow automation issues by downloading and reviewing log files from the File Delivery Status page.

File Delivery Status page

The File Delivery Status page provides access to original and transformed files, as well as information and error logs to help troubleshoot file delivery, processing and loading issues. The grid displays the following columns:
Original Filename
The filename of the original source file.
Transformed Filename
For files that are loaded and transformed through a connector such as IBM® API Connect, the transformed filename.
Delivered On
The date and time the file was shared by the client on Amazon S3.
Delivered From
The data service that was used to process the pipeline.
Delivered Via
The data service type that delivered the data, for example, Amazon S3.
Delivery Status
An indication of whether the file downloaded successfully from the data source or whether the data source encountered an error.
Processing Status
An indication of whether the file was successfully loaded and transformed, whether the file was successfully pushed to App Connect but not yet transformed, or an indication if the transmission or the transformation encountered an error.
Loading Status
An indication of whether the file loaded successfully or whether an error was encountered. A loading status value of Loaded with errors indicates that some of the rows in a file were not loaded and you should check the load errors in the File Loading Logs page.
Skipped rows
An indication of the number of rows that were skipped in the file.
Parse errors
An indication of the number of parse errors with warning and error severity levels. The number excludes informational and debug error messages.
Load errors
An indication of the number of load errors that occurred for the file.
Alerts
An indication of the number of alerts that occurred for the file.

Downloading files

Note: You can download a copy of an original or transformed file only if the file has been processed and validated to contain data for your organization. If the download action fails because of an organization ID validation error, contact IBM support.
To download a copy of an original or transformed file, or to download multiple files in bulk, complete the following steps. If you select multiple files for download, the total download size is limited to 100 MB.
  1. As a system administrator, from the main menu, click Admin > Data Flow Automation to open the Data Services grid.
  2. Click the File Delivery Status tab.
  3. Choose one of the following options:
    • To download a copy of an original source file, select one or more files, click Actions and select Download Original File(s).
    • To download a copy of a transformed file, select one or more files, click Actions and select Download Transformed File(s). This option is relevant only for files that pass through a custom pipeline with transformation.

Reviewing file loading logs

You can view the File Loading Logs page to see more details about a file that has loaded or failed to load. The File Loading Logs grid now includes columns that show the number of skipped rows, non-informational parse errors, load errors, and alerts. A file status value of Loaded with errors indicates that some of the rows in a file were not loaded.

To view the file loading logs, complete these steps:
  1. As a system administrator, from the main menu, click Admin > Data Flow Automation to open the Data Services grid.
  2. Click the File Delivery Status tab.
  3. Select a file, click Actions and select Loading Logs.
  4. Review the loading status of the file by checking the Result, Compilation Status, Record In and Records Out values.
  5. Select the file, click Actions and select Parse Errors to view parse errors.
  6. Select the file, click Actions and select Load Errors to view load errors.
For more information about reviewing file loading logs, refer to Troubleshooting uploaded files.

Reviewing data delivery log files

You can view the file delivery logs to check for issues when the Delivery Status or Processing Status shows an error.

To view the delivery logs, complete these steps:
  1. As a system administrator, from the main menu, click Admin > Data Flow Automation to open the Data Services grid.
  2. Click the File Delivery Status tab.
  3. Select a file, click Actions and select Delivery Logs.
  4. Review the log entries to check for errors. Common errors include files that match no pipelines or multiple pipelines, and files that fail to transform via App Connect.
  5. Review additional log details, including source and destination of the files in the Preview panel.

Deleting records

A system administrator can delete the records that are loaded from an account data file.

To delete a file, complete the following steps:
  1. As a system administrator, from the main menu, click Admin > Data Flow Automation to open the Data Services grid.
  2. Click the File Delivery Status tab.
  3. Select a file, click Actions and select Loading Logs.
  4. Select a file, and click Actions > Delete Records.