Monitoring data for file ingestion

After you upload data to your bucket for a data type, Sterling Intelligent Promising polls the data file to begin ingestion. To help ensure that file ingestion is successful, you must monitor the failed prefix for your storage bucket and look for any data that contains errors. Errors can occur due to incorrect file format, missing data, or connectivity issues. Evaluate the execution report from the summary prefix to understand the reason for failure.

Before you begin

Transfer the data to your bucket. For more information, see Transferring data by using the Sterling Intelligent Promising file transfer adapter.

Understand the file prefixes that are created in Sterling Intelligent Promising. For more information, see File ingestion prefix structure.

Sterling Intelligent Promising validates the data to help ensure that any invalid data sets are eliminated. Become familiar with the data ingestion errors that you might encounter and learn to troubleshoot them. For more information, see Troubleshooting data ingestion errors.

About this task

To help ensure that file ingestion is successful, you must monitor the failed prefix for your storage bucket and look for any data that contains errors.

Procedure

  1. Monitor the failed prefix and ensure that you are set up to receive notifications for any failed files. You can build an event monitoring service or use the event notifications feature that is available for each storage service provider.
  2. Download both the failed rows CSV file and the failure reasons text file and locate the problematic data in the failed rows CSV file.
  3. Review the issues that are listed in the failure reasons text file and locate the problematic data in the failed rows CSV file.
  4. Correct the data as needed.
  5. Re-upload the corrected CSV file to the original source prefix for reprocessing.