Job log retention in DataStage

Modify the log retention file on the px-runtime persistent volume to set log cleanup policy.

Every job run generates log files on the runtime instance PV. The filepath to the logs for a particular job run is /ds-storage/PXRuntime/Projects/<projectID>/jobs/<jobID>/runs/<jobRunID>. The generated files include a job log, monitor data, and job parameters.

The log retention policy handles cleanup of job log files on the PV. It deletes old log files after a set number of days or runs. Allowing lots of log files to build up can reduce performance. Configure log retention policy for a specific runtime instance by modifying the config file under /px-storage/config/log_retention. Enable or disable cleanup, determine whether to delete files after a set number of days or runs, and set frequency for cleanup to run.