Implementing log retention policy

To manage cleanup and retention of logs, kernel directories, and project spaces, configure the following components:

Analytics Engine Custom Resource (ae-cr) – Log retention configuration

To enable automatic cleanup of instance and engine logs (default: 30 days), add the retention_period section to the Analytics Engine Custom Resource:


spec:
  autoScaleConfig: false
  blockStorageClass: managed-nfs-storage
  cloudpakfordata: true
  fileStorageClass: managed-nfs-storage
  image_digests:
    spark-hb-helm-repo: sha256:1b9512f8b2717cccb2f92f60218847d1561279051b8b9d1b641dbd152e2e1121
  license:
    accept: true
  retention_period:
    engine_duration: <number-of-days>
    instance_duration: <number-of-days>

Kernel Directory Retention – ConfigMap Update

The administrator must specify the number of kernel directories to retain by editing the spark-hb-logs-cleanup-policy ConfigMap. By default, it uses a 30-day retention policy:

apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-hb-logs-cleanup-policy
data:
  logs_cleanup_config.json: |
    {
      "project_retention_days": 30,
      "space_retention_days": 30,
      "git_project_retention_days": 30
    }

To customize retention duration, update the values:


logs_cleanup_config.json: |
{
  "project_retention_days": <number-of-days>,
  "space_retention_days": <number-of-days>,
  "git_project_retention_days": <number-of-days>
}

Learn more

For specifying additional configurations for Analytics Engine powered by Apache Spark, see additional configurations.