Data purger configuration

Edit the data purger configuration files to specify settings for data purging, such as enabling archiving before data is purged.

After editing each configuration file, restart the purger service for your changes to take effect.

Duration to store records before purging

Specify the duration that records should be stored in the database, before being purged
To specify the duration for records to be stored before being purged, edit the data purger configuration filesfile:
  • Linux®:
    • $EGO_CONFDIR/../../perf/conf/purger/purger_ego.xml (EGO)
    • %EGO_CONFDIR/../../perf/conf/purger/purger_soam.xml (IBM® Spectrum Symphony)
    • %EGO_CONFDIR/../../perf/conf/purger/purger_pmr.xml (MapReduce)
  • Windows:
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_ego.xml (EGO)
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_soam.xml (IBM Spectrum Symphony)
Specify the duration, using the following syntax:
<TableList Duration=expiry_time>

where expiry_time is the expiry time, in days, up to a maximum of 365 days, which is calculated from the date of the last record in the database.

For example, if the first record is dated July 1, the last record is dated July 20, and the duration is set to 10 days, the records between July 10 and July 20 are retained while the records between July 1 and July 10 are deleted.

This default expiry time applies to all tables, but may be overridden for individual tables by specifying a record expiry time for that table.

Specify the duration that records, in a specific table, should be stored in the database, before being purged
To specify the duration that records, in a specific table, are stored before being purged, edit the data purger configuration files:
  • Linux:
    • $EGO_CONFDIR/../../perf/conf/purger/purger_ego.xml (EGO)
    • $EGO_CONFDIR/../../perf/conf/purger/purger_soam.xml (IBM Spectrum Symphony)
    • $EGO_CONFDIR/../../perf/conf/purger/purger_pmr.xml (MapReduce)
  • Windows:
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_ego.xml (EGO)
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_soam.xml (IBM Spectrum Symphony)
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_pmr.xml (MapReduce)
Specify the duration, using the following syntax:
<Table TableName=table_name ... Duration=expiry_time>
where:
  • table_name is the name of the individual table
  • expiry_time is the expiry time, in days, up to a maximum of 365 days, which is calculated from the date of the last record in the database.

    For example, if the first record is dated July 1, the last record is dated July 20, and the duration is set to 10 days, the records between July 10 and July 20 are retained while the records between July 1 and July 10 are deleted.

This expiry time overrides the default expiry time for all records as specified in the <TableList> element.

Specify the duration to keep data used for the Explorer Charge Back chart in the Elasticsearch tables, before being purged
By default, Charge Back chart data is kept in the Elasticsearch database for 14 days before being purged. To customize this value, edit the Elasticsearch services data purger configuration file: $EGO_CONFDIR/../../integration/elk/conf/indexcleanup/sym_explorer.conf.
For example, to extend the duration from 14 days to 30 days, within the configuration file, add 30 after the Elasticsearch database index name (ibm-symphony-report-charge-back), using the following syntax:
ibm-symphony-report-charge-back* 30

After making the changes to the configuration file, restart the elk-manager service for the changes to take effect.

Archiving data before purging

Enable archiving of old data from a specific table before being purged
By default, the data purger does not archive old data from a specific table before purging. You can enable archiving in the purger configuration files:
  • Linux :
    • $EGO_CONFDIR/../../perf/conf/purger/purger_ego.xml (EGO)
    • $EGO_CONFDIR/../../perf/conf/purger/purger_soam.xml (IBM Spectrum Symphony)
    • $EGO_CONFDIR/../../perf/conf/purger/purger_pmr.xml (MapReduce)
  • Windows:
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_ego.xml (EGO)
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_soam.xml (IBM Spectrum Symphony)
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_pmr.xml (MapReduce)
To enable data archiving, change the value from <Table TableName=table_name ... Archive=false> to <Table TableName=table_name ... Archive=true>, as follows:
<Table TableName=table_name ... Archive=true>

where table_name is the name of the individual table.

Disable archiving of old data from a specific table before being purged
By default, the data purger does not archive old data from a specific table before purging. If you changed the behavior to enable archiving, and want to set it back to the default setting, do this in each data purger configuration files:
  • Linux:
    • $EGO_CONFDIR/../../perf/conf/purger/purger_ego.xml (EGO)
    • $EGO_CONFDIR/../../perf/conf/purger/purger_soam.xml (IBM Spectrum Symphony)
    • $EGO_CONFDIR/../../perf/conf/purger/purger_pmr.xml (MapReduce)
  • Windows:
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_ego.xml (EGO)
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_soam.xml (IBM Spectrum Symphony)
    • %EGO_CONFDIR%\..\..\perf\conf\purger\purger_pmr.xml (MapReduce)
To disable data archiving, change the value from <Table TableName=table_name ... Archive=true> to <Table TableName=table_name ... Archive=false>, as follows:
<Table TableName=table_name ... Archive=false>

where table_name is the name of the individual table.

Purge time and interval

Specify the time in which the data purger purges old data daily
Use the cluster management console to configure the data purger service profile, and set the profile to include this setting: 
<ego:Command>... file_path/purger... -t purge_time
where purge_time is the 24-hour daily time, in hh:mm format.
Specify the time interval in which the data purger purges old data, starting from when the purger service first starts up
Use the cluster management console to configure the data purger service profile, and set the profile to include this setting:
  • To purge old data at a regular interval in hours:
    <ego:Command>... file_path/purger... -t *[purge_time_interval]
    where purge_time_interval is the time interval in hours.
  • To purge old data at a regular interval in minutes:
    <ego:Command>... file_path/purger... -t *:*[purge_time_interval]
    where purge_time_interval is the time interval in minutes.

Automatically starting the purger service

Enable automatic startup of the purger service
By default, automatic startup of the purger service is enabled. If you changed the behavior to disable automatic startup, and want to set it back to the default setting, use the cluster management console to configure the data purger service profile, and set the profile to include the <sc:StartType>AUTOMATIC</sc:StartType> setting.
Disable automatic startup of the purger service
By default, automatic startup of the purger service is enabled. To disable automatic startup, use the cluster management console to configure the data purger service profile, and set the profile to include the <sc:StartType>MANUAL</sc:StartType> setting.

Log file

Specify the default log level of your data purger log file
To specify the default log level of your data purger log file, edit the log4j.properties file:
  • Linux: $EGO_CONFDIR/../../perf/conf/log4j.properties
  • Windows: %EGO_CONFDIR%\..\..\perf\conf\log4j.properties
Specify the default log level, using the following syntax:
log4j.logger.domain_name.perf.dataloader=log_level, domain_name.perf.purger

where log_level is the default log level of your data purger log file.

The data purger only logs messages of the same or less detail as log_level. Therefore, if you change the log level to ERROR, the loader controller will only log ERROR and FATAL messages.

Once you have saved your changes to the configuration files, restart the purger service for your changes to take effect.