Automating data imports

You can upload your data on a periodic basis into a object storage bucket by using CSV or JSON files. The Supply Chain Intelligence Suite administrator creates import rules to import data at predefined intervals from your object storage bucket.

To automate your data import, complete the following steps:
  1. Review the data dictionary and prepare your CSV or JSON files. Ensure that your data files are formatted correctly in CSV or JSON format. For more information, see Sample CSV files or Sample JSON data.
    Note: The last column (Used as identifier (default csv schema)) of the data dictionary is not applicable for the JSON file import.
  2. Analyze your data and determine what data models you need to use for your business case.
  3. If you do not already have object storage, provision your object storage bucket where you upload data. For more information, see Managing object storage.
    Note: To automate data ingestion by using an object storage bucket, you need to create a custom bucket with full access permissions. The default bucket included with the Supply Chain Intelligence Suite has limited functions and access, and is suitable for most of the basic needs.
  4. Create an import rule by using rule builder within Supply Chain Intelligence Suite to process your data from the defined object storage bucket. For more information, see Creating import rules.
  5. Upload your CSV or JSON files to object storage using the appropriate API for your object storage instance. For more information, see Ingesting CSV data or Ingesting JSON data.
  6. Ensure that the object storage upload returned 200 status code to indicate success and that the object is uploaded to your object storage instance. Once the file has been read and processed by Supply Chain Intelligence Suite, a JSON status file with the format of YOURFILENAME_READDATE_status.json, for example testFile_2024-05-01T220611Z_status.json is created in the importStatus directory of the object storage bucket.
  7. Optional: To determine when the data uploaded to COS has been imported and processed by Supply Chain Intelligence Suite the status of the Supply Chain Intelligence Suite job is required. To access the status of the job, either use the Job Manager API or read and parse the JSON status file created in the importStatus directory of the object storage bucket. The JSON status file can be accessed using the object storage API.