Scaling Elasticsearch replicas to provide redundant copies of data to protect against hard failure

By default, Elasticsearch indices are configured with one primary shard and one replica at index creation. This configuration can tolerate a failure of at most one host while Elasticsearch migrates shards to rebalance the cluster. You can scale up the default Elasticsearch replicas to increase redundant copies of your data.

Before you begin

  • Increasing the replicas also increase the disk usage. For more information about preventing full disk or high watermark issues, see Configuring Elasticsearch disk usage.
  • The procedure modifies the index at index creation. The changes will apply to indices created the day after the changes are made as the indices for the current day is created.
  • The maximum replicas that can be configured is N-1, where N is the number of hosts in the resource group the Elasticsearch data service is configured to use.

Procedure

  1. Stop the Elastic Stack services. For more information, see Stopping Elastic Stack services.
  2. Open in a text editor the $EGO_CONFDIR/../../integration/elk/init/template/ibm-symphony-report-charge-back.json configuration file.
    Remember: For local installation, you must complete this step on all management hosts in your cluster.
    Replace the value in "number_of_replicas" : 1 with the number of replicas that you want to increase. For example, the replicas in the following configuration are modified to two replicas to tolerate a failure of two hosts:
    {
      "index_patterns": "ibm-symphony-report-charge-back*",
      "order": 1,
      "settings": {
          "index": {
            "number_of_shards": 1,
             "number_of_replicas" : 2
          }
       },
       "mappings": {
       			"properties" :{
                    "Type": { "type": "keyword" },
       				"CHARGEBACK_VALUE":{ 
       					"type":"double"
       				},
       				"EVENT_TIME":{ 
       					"type":"date"
       				},
       				"CLUSTER_NAME":{ 
       					"type":"keyword"
       				},
       				"CONSUMER_NAME":{ 
       					"type":"keyword"
       				},
       				"RESOURCE_GROUP":{ 
       					"type":"keyword"
       				}
       			},
       		"date_detection" : true,
       		"numeric_detection" : false,
       		"dynamic_templates":[{
       			"str":{ 
       				"match": "*",
       				"match_mapping_type": "string",
       				"mapping":{
       					"type": "keyword"
       				}
       			}
       		}]
       } 
  3. Restart the Elastic Stack services. For more information, see Starting Elastic Stack services.