Creating Rollover of an Index

Edit online

About this task

When data on a particular index exceeds a certain limit, it is essential to rollover the index and create a new index. In API Gateway, it is essential to monitor the indexes for transactional events. For transactional events, you must rollover the index, when the index size exceeds 25 GB. When an index is rolled over, a new index is created with two primary and a replica for each shard.
You can perform rollover for the following indexes:
  • performancemetrics
  • policyviolationevents
  • monitorevents
  • errorevents
  • threatprotectionevents
  • transactionalevents
  • lifecycleevents
  • auditlogs
  • log
  • serverlogtrace
  • mediatortrace
  • requestresponsetrace

To create rollover of an index

Procedure

  1. From the command prompt, go to SAGInstallDir/IntegrationServer/instances/default/packages/WmAPIGateway/cli/bin.
    Note: Replace default with the corresponding instance name.
  2. Run the following command to create a backup of specific indexes in the default location:
    Linux 
    ./apigatewayUtil.sh rollover index -type name of the index to roll over
    Windows 
    apigatewayUtil.bat rollover index -type name of the index to roll over

    The following sample shows the roll over of index, performancemetrics, on Windows. 

    C:\SoftwareAG\IntegrationServer\instances\default\packages\WmAPIGateway\cli\bin>apigatewayUtil.bat rollover index 
	-type performancemetrics

Rollover is completed successfully
Old Index: gateway_default_analytics_performancemetrics_202108140608
New Index: gateway_default_analytics_performancemetrics_202108140612
Rolled over: true
Dry Run: false
Conditions: [{}]

    You can include the following optional parameters to customize the roll over command as per your requirements.

    Parameter Description
    tenant Name of the tenant in which the data exists.

    If you do not provide this parameter, the value is picked from pg.gateway.elasticsearch.tenantId property in config.properties file located at IntegrationServer\instances\instance_name\packages\WmAPIGateway\config\resources\elasticsearch.

    targetIndexSuffix Suffix to be included in the name of the rolled over index. If you do not provide this parameter, the rollover timestamp is added as a suffix to the rolled over index. You can provide a suffix such as the date distinguish the index.
    For example, if you provide the following command: 
    apigatewayutil.bat rollover index 
-type performancemetrics -targetIndexSuffix 15aug2021
    the suffix is added to the rolled over index 
    Rollover is completed successfully
Old Index: gateway_default_analytics_performancemetrics_14aug2021
New Index: gateway_default_analytics_performancemetrics_15aug2021
Rolled over: true
Dry Run: false
Conditions: [{}]
    dryRun Option to check whether the current index matches with any of the specified conditions.
    Possible values are:
    • true. Performs check on the current index against the specified conditions.
    • false. Does not check the current index. This is the default value and the value of the parameter is considered as false, if do not specify this parameter.
    You can specify one of the conditions listed in the Conditions that can be given with rollover command section.
    For example, to check if the size of the current log index is 6 GB, you can run the command as seen here: 
    apigatewayutil.bat rollover index -type 
log -dryRun true -maxSize 6GB
    Sample result: 
    Dry run executed successfully.
Old Index: gateway_default_log_1628582905777-000001
New Index: gateway_default_log_202108271018
Rolled over: false
Dry Run: true
Conditions: [{[max_size: 6gb]=false}]
    Note: This parameter does not perform rollover. It displays results based on the check performed.
    logLevel Level of log that you want to create. Log levels indicate the severity of logs. Available levels are:
    • Info. Provides the list of regular events that occur during the process. These events are informative.
    • Debug. Provides the events that could be useful, if you have to debug the process.
    • Warning. Indicates unexpected events that occurred during the process. Usually, these events do not interrupt or have an immediate effect on the process.
    • Error. Indicates the events that stop the functionality from working as designed.
    • Trace. Provides the list of events in a much detailed manner that could be useful for debugging.
    You can specify one of the log level with the logLevel parameter. For example, to create a log file of Warning level when listing backup files, you can run the following command: 
    apigatewayUtil.bat list backup -logLevel warning
    When you provide Error as the log level, then only the error level logs are saved. When you provide Debug as the log level, then Debug, Info, Warning and Error level logs are saved. When you provide Trace as log level, then all level logs are saved.

    This parameter is optional. If you do not specify the parameter, then the Info level logs are saved by default.

    logFileLocation Location where you want to save the log file.
    For example, to save the log file in C:/apiglogs/backups, you can provide the location as seen in the following example: 
    apigatewayUtil.bat create backup -name samplebackup 
-logFileLocation C:/apiglogs/backups
    This parameter is optional. If you do not specify the parameter, the logs are saved in the following location SAGInstallDir/IntegrationServer/instances/instance_name/packages/WmAPIGateway/cli/logs/APIGWUtility.log.
    The specified indexes are rolled over when any one of these conditions is satisfied.

Conditions that can be given with rollover command

Edit online

About this task

In addition to the above parameters you can provide the following conditions based on which the specified indexes must be rolled over:
Condition Description
maxAge Maximum age of the indexes.

The specified indexes are rolled over if they are older than the value provided for this condition. For the list of possible values for this field, see https://www.elastic.co/guide/en/elasticsearch/reference/master/api-conventions.html#time-units.

For example, to roll over the monitorevents index if it is older than 2 minutes, you can provide 
apigatewayutil.bat rollover index 
-type monitorevents -maxAge 2m
maxDocs Maximum number of the documents in the indexes.

The specified indexes are rolled over if the number of documents in the indexes are more than or equal to the value provided for this condition.

For example, to roll over the monitorevents index if the number of documents in the index is more than or equal to 100, you can provide 
apigatewayutil.bat rollover index 
-type monitorevents -maxDocs 100
maxSize Maximum size of the indexes.

The specified indexes are rolled over if their size is equal to or more than the value provided in this condition. For the list of possible values for this field, see https://www.elastic.co/guide/en/elasticsearch/reference/master/api-conventions.html#byte-units.

For example, to roll over the monitorevents index if its size is more than 1 GB, you can provide 
apigatewayutil.bat rollover index 
-type monitorevents -maxAge 1gb
maxPrimaryShardSize Maximum size of the primary shard of the indexes.

The specified indexes are rolled over if the size of their primary shards is equal to or more than the value provided for this condition.

For example, to roll over the monitorevents index if the size of the index's primary shard is more than or equal to 1 GB, you can provide 
apigatewayutil.bat rollover index 
-type monitorevents -maxPrimaryShardSize 1GB