Modifying the data retention policy for logging services

Logging service data is persisted on disk. Over time, unmanaged data growth fills up your disk space. To keep data size under control, retention policies are defined. The logging service curator component cleans up log data based on your retention policy.

In addition to the automatic removal of log data, you can manually remove log indices. For more information, see Manually removing log indices

Before you begin, consider the following tips:

• curator runs on a daily schedule. As a result, you might retain one extra day of data.
• When the data index is removed, a new data index is re-created when new data is sent to it.
• Increased retention periods require more disk space and other compute resources. You might experience stability issues when you define longer retention periods without adjusting your resources.

View log indexes and the default retention policies for a logging instance in the following table.

Complete the following steps to customize the data retention policy.

Note: If you follow the procedure to update the chart values, add the same lines that you include in the values-override.yaml file to your config.yaml file. This method allows the installer to reapply the settings changes during upgrade and rollback operations. Upgrade resets the chart to the chart defaults, overridden by the values set in your config.yaml file.

1. Extract the existing logging service (Helm release) parameters.

   • Extract Helm parameters by running the following command:

     helm get values logging_service_release_name --tls > values-old.yaml
     

     For example:

     helm get values logging --tls > values-old.yaml
     

     Note: logging is the Helm release name of the logging service installed by your product installer.

   • Optionally, apply prior adjustments. All Kubernetes resource manifest adjustments that are made by using the kubectl command are overridden with values that are defined in chart parameters. Replica count, JVM heap size, or container memory limits are a few examples. If prior Kubernetes resource manifests were adjusted, make sure that you apply the same adjustments to values-old.yaml.

2. Prepare chart parameters.

   • Create a values-override.yaml file to include the following curator settings.

     Note:

     • app refers to the container log.
     • You can set the unit value to values other than days.

     • Avoid setting a retention policy to less than one day.

       curator:
        # in this example, container log retention period is set to 2 days
        app:
          count: 2
          unit: days
       

       For detailed parameter information, see the Helm chart readme file.

3. Download the chart.

   • Identify chart version.

     Logging chart versions vary based on the installed IBM Cloud Private version. You can use the console to find chart versions in the service catalog. The logging chart can be identified by the name ibm-icplogging under the mgmt-repo repository. You can also select SOURCE & TAR FILES from the console to find a local link to a chart.

   • Download the chart .tar file.

     Run the following command by using the local link found in Step 3:

      curl -k https://<Cluster Master Host>:<Cluster Master API Port>/mgmt-repo/requiredAssets/ibm-icplogging-x.y.z.tgz > ibm-icplogging-x.y.z.tgz
     

     For more information, see Accessing your cluster by using the console.

4. Upgrade the Helm chart. Run the following command. Replace x.y.z with the version found in Step 3.

    helm upgrade logging_service_release_name ibm-icplogging-x.y.z.tgz -f values-old.yaml -f values-override.yaml --force --timeout 600 --tls
   

5. The logging service becomes available in approximately 5 - 10 minutes. You can also check Helm upgrade status by using the following command:

    helm history --tls logging_service_release_name