Deleting data that is no longer needed

To manage data that was imported into the Z Data Analytics Platform but is no longer needed, a utility script cleanZdap.sh is available in the <ZOA_HOME>/bin directory. This utility can be used to perform a one-time cleanup of data that is no longer needed, or to support regularly scheduled purging of data that exceeds a configured period.

About this task

This utility deletes indexes containing ingested data based on the timestamp contained in the index name. It does not support any additional filters such as data source type or the name of the system from which the data was collected.

Procedure

  1. To configure the data cleanup utility, set the value of the ZDAP_DATASTORE_RETENTION_PERIOD property in the zoa_env.config file to the wanted retention period. Any indexes with a timestamp older than the specified retention period are deleted when the utility is run.
    The retention period can be specified in days or weeks. However, since indexes are created on weekly boundaries, the granularity of data deletion is also weekly. Examples of valid entries include:
    • ZDAP_DATASTORE_RETENTION_PERIOD="4 weeks": Any indexes older than 4 weeks are deleted when the utility is run.
    • ZDAP_DATASTORE_RETENTION_PERIOD="3 days": Any indexes older than 1 week are deleted when the utility is run.
  2. To run the data cleanup utility, run the following commands under the same user ID that is used to run the Z Data Analytics platform: 
    cd <ZOA_HOME>
./bin/cleanZdap.sh
    The utility prints the configured retention period and each index that it deletes.
  3. To run the data cleanup utility on a schedule, complete the following steps under the same user ID that is used to run the Z Data Analytics Platform:
    1. Edit the crontab definition for the user ID:
      crontab -e
    2. Append lines similar to the following to the crontab definition. The example below enables the cleanZdap.sh utility to run every night at midnight:
      # cleanZdap.sh: Delete ZDAP indices older than a configured time offset
0 0 * * * <ZOA_HOME>/bin/cleanZdap.sh
    3. Save and close the crontab definition.
    4. Validate that the crontab definition is correctly created:
      crontab -l