Changing the index purge duration for Elasticsearch

Edit online

To avoid unchecked growth of data stored in Elasticsearch, older data must be deleted periodically. This is accomplished by rotating indexes and deleting them after the predefined retention period.

About this task

The script used to delete expired indexes is installed on the IBM Spectrum LSF Explorer Server host under /etc/cron.daily/drop_es_indices and will automatically be executed daily.

The configuration file indexpurge.conf is used to configure the index retention period and can be edited to customize the retention period.

Note: Each index name must follow the pattern "<prefix>-<date>", where "<prefix>" is determined by the index contents and "<date>" is the index creation date in either of the following two formats: YYYY.MM.DD or YYYYMM.

Procedure

  1. Log on to the IBM Spectrum LSF Explorer Server host.
  2. Open the SERVER_TOP/lsfsuite/ext/explorer/server/config/indexpurge.conf file, where SERVER_TOP is the IBM Spectrum LSF Explorer Server installation location.
  3. Modify the retention days of the index you want to change.

    For example, to save disk space, reduce the indices lsf_jobs_active-[YYYYMM] and lsf_jobs_pend-[YYYYMM] data period from 60 days to 30 days:

    Change Lines:
    lsf_jobs_active- 60
lsf_jobs_pend- 60
    to
    lsf_jobs_active- 30
lsf_jobs_pend- 30