Sample ingest utility scripts

You can use the ingest utility sample script to automate writing a new INGEST command each time there are new files to process.

The sample script ingest_files.sh is a shell script that automatically checks for new files and generates an INGEST command to process the files. The script performs the following tasks, in order:
  1. Check the directory to see if there are new files to process. If there are no files, the script exits.
    Note: The script assumes that the specified directory only contains files for the table that you want to populate.
  2. Obtain the names of the new files and then generate a separate INGEST command for each file
  3. Run the INGEST command and handle the return code
  4. Move the processed files to a success directory or a failed directory.

The script is provided in the samples/admin_scripts directory under your installation directory.

Modifying the script for your environment

You can use the ingest_files.sh script as a basis for your own script. The important modifications that you have to make to it are:
  • Replace the sample values (namely, the database name, table name) with you own values
  • Replace the sample INGEST command with your own command
  • Create the directories specified in the script
The script processes files that contain data to populate a single table. To populate multiple tables, you can either replicate the mechanism for each table that you want to populate or generalize the mechanism to handle multiple tables.

Sample scenario

A sample scenario has been included in the documentation to show you how you can adapt the sample script to your data warehouse to automate the generation of new INGEST commands.