Modifying the data retention policy for logging services

Logging service data is persisted on disk. Over time, unmanaged data growth fills up your disk space. To keep data size under control, retention policies are defined. The logging service curator component cleans up log data based on your retention policy.

In addition to the automatic removal of log data, you can manually remove log indices. For more information, see Manually removing log indices

Before you begin, consider the following tips:

View log indexes and the default retention policies for a logging instance in the following table.

Table 1. Logging service indexes
Log Index Description Chart Parameter Default Retention Period (day)
logstash-* Container logs 1
audit-* audit logs curator.auditLog 1
compliance-* Vulnerability Advisor data 90
secconfig1-* Vulnerability Advisor data 90
vulnerabilityscan-* Vulnerability Advisor data 90
mapp-raw-* Mutation Advisor data curator.mapp 2
mapp-events-* Mutation Advisor data curator.mapp 2
mapp-accum-* Mutation Advisor data curator.mapp 2
sas_info-compliance-* Vulnerability Advisor data curator.mapp 90
sas_info-secconfig1-* Vulnerability Advisor data curator.mapp 90
sas_info-vulnerabilityscan-* Vulnerability Advisor data curator.mapp 90

Complete the following steps to customize the data retention policy.

Note: If you follow the procedure to update the chart values, add the same lines that you include in the values-override.yaml file to your config.yaml file. This method allows the installer to reapply the settings changes during upgrade and rollback operations. Upgrade resets the chart to the chart defaults, overridden by the values set in your config.yaml file.

  1. Extract the existing logging service (Helm release) parameters.

    • Extract Helm parameters by running the following command:

      helm get values logging_service_release_name --tls > values-old.yaml

      For example:

      helm get values logging --tls > values-old.yaml

      Note: logging is the Helm release name of the logging service installed by your product installer.

    • Optionally, apply prior adjustments. All Kubernetes resource manifest adjustments that are made by using the kubectl command are overridden with values that are defined in chart parameters. Replica count, JVM heap size, or container memory limits are a few examples. If prior Kubernetes resource manifests were adjusted, make sure that you apply the same adjustments to values-old.yaml.

  2. Prepare chart parameters.

    • Create a values-override.yaml file to include the following curator settings.


      • app refers to the container log.
      • You can set the unit value to values other than days.
      • Avoid setting a retention policy to less than one day.

         # in this example, container log retention period is set to 2 days
           count: 2
           unit: days

        For detailed parameter information, see the Helm chart readme file.

  3. Download the chart.

    • Identify chart version.

      Logging chart versions vary based on the installed IBM Cloud Private version. You can use the console to find chart versions in the service catalog. The logging chart can be identified by the name ibm-icplogging under the mgmt-repo repository. You can also select SOURCE & TAR FILES from the console to find a local link to a chart.

    • Download the chart .tar file.

      Run the following command by using the local link found in Step 3:

       curl -k https://<Cluster Master Host>:<Cluster Master API Port>/mgmt-repo/requiredAssets/ibm-icplogging-x.y.z.tgz > ibm-icplogging-x.y.z.tgz

      For more information, see Accessing your cluster by using the console.

  4. Upgrade the Helm chart. Run the following command. Replace x.y.z with the version found in Step 3.

     helm upgrade logging_service_release_name ibm-icplogging-x.y.z.tgz -f values-old.yaml -f values-override.yaml --force --timeout 600 --tls
  5. The logging service becomes available in approximately 5 - 10 minutes. You can also check Helm upgrade status by using the following command:

     helm history --tls logging_service_release_name