Adding a node to or removing a node from an existing Elasticsearch cluster

You can add nodes to any Elasticsearch cluster. Removing a node is possible only if the cluster has at least four nodes.

Before you begin

To ensure that the cluster works correctly and all nodes are available, enter

curl -X GET 'http://primary_node_ip:9200/_cluster/state?pretty' 
curl -X GET 'http://primary_node_ip:9200/_nodes?pretty'

Where primary_node_ip is the IP address of either the single Elasticsearch node or the primary Elasticsearch node in a cluster of at least three nodes.

Adding a node

About this task

You can add a single node to any Elasticsearch cluster with three or more nodes. With a single node deployment, you must add at least two nodes as to create a default three node Elasticsearch cluster.

Procedure

Download the Elasticsearch OVA version that matches the current cluster version from Fix Central and deploy it to vCenter.
In VMware vSphere Client, check the VM settings that were created from the OVA and change them if required.
Primarily, check these settings:

Memory

Must be at least 32 GB

Hard disk 1 (primary disk)

Must be at least 100 GB

Hard disk 2 (data disk)

Must be at least 1 TB

Open the console to launch the configuration wizard (first-boot procedure).

Depending on your version of vSphere Client, you might either have a Launch Console or an Open Console link to do so.

The wizard is text based. To work in the wizard, use the Tab key to navigate, the Space bar to select items, and the Enter key to apply your selections.

Accept the CentOS and IBM Eula license agreements when prompted.
Set up the passwords for the root and builder accounts.
With the builder account, you can later log in to the virtual machine via SSH.

Configure the network.

Within the Network Configuration window, select either Static or Obtain IP via DHCP.

If you select the Static IP option, complete these fields to configure the static address:

Parameter	Value
Hostname	The fully qualified host name of the Elasticsearch node
IP Address	The IPv4 address of the Elasticsearch node
Netmask	The netmask for the assigned IP address
Gateway	The IP address of the default gateway for the IP subnet Note: This is the network gateway, not the IBM® StoredIQ® gateway.
Primary DNS Host	The IP address for the domain name server

If you select the Obtain IP via DHCP option, provide this information:
Parameter Value

Hostname The fully qualified host name of the Elasticsearch node

Also select Restart network services to restart the node after your configuration is complete.

Log in to the Elasticsearch node, where the installation was initially started, by using the builder account and password that were configured during the installation of the cluster.
Edit the cluster-setup.properties file by appending the new node’s IP address to the ES_HOSTS line. For example, to add a fourth node to a three node cluster, the ES_HOSTS line looks like:

ES_HOSTS=192.0.2.0,192.0.2.10,192.0.2.24,198.51.100.0

where 198.51.100.0 is the new node's IP address.
Run the cluster setup script. At the prompt [builder@localhost ~]$, enter:

/siq/bin/cluster-setup.sh cluster-setup.properties
Accept the server certificate and enter the password for the new node when prompted.
After the script is executed successfully, the new node is part of the Elasticsearch cluster.
Add the new node's IP address and port information to the /usr/local/storediq/etc/siq-elasticsearch.yml file on the data servers that use this Elasticsearch cluster.
Your .yml file should look similar to the example where 198.51.100.0 is the new node's IP address:
```
cluster:
 name: Elasticsearch
nodes:
- host: 192.0.2.0
  port: 9200
- host: 192.0.2.10
  port: 9200
- host: 192.0.2.24
  port: 9200
- host: 198.51.100.0
  port: 9200
```
Enter the command to restart services on the data servers that are using this Elasticsearch cluster:
```
service deepfiler restart
```

To ensure that the cluster works correctly and all nodes are available, enter:

curl -X GET 'http://primary_node_ip:9200'
curl -X GET 'http://primary_node_ip:9200/_cluster/health?pretty=true'
curl -X GET 'http://primary_node_ip:9200/_cluster/state?pretty'

Removing a node

About this task

You can remove nodes only from clusters with at least four nodes.

Procedure

Log in to the primary node, where the installation was initially started, by using the builder account and password that were configured during the installation of the cluster.

Decommission a node by entering:

curl -XPUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "node_ip"
   }
}';

where node_ip is the IP address of the node that is being decommissioned.

The response must look like this:

{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":
{"allocation":{"exclude":{"_ip":"node_ip"}}}}}}[builder@localhost ~]$

Edit the cluster-setup.properties file and remove the decommissioned node's IP address from the ES_HOSTS line.
Run the cluster setup script. At the prompt [builder@localhost ~]$, enter:

/siq/bin/cluster-setup.sh cluster-setup.properties
Accept the server certificate and enter the password for each node when prompted.
Enter the following commands to ensure that the cluster works correctly and all nodes are available (the decommissioned node is still listed, but no shards are allocated to it):
```
curl -X GET 'http://primary_node_ip:9200'
curl -X GET 'http://primary_node_ip:9200/_cluster/health?pretty=true'
curl -X GET 'http://primary_node_ip:9200/_cluster/state?pretty'
```
Shut down and delete the VM in vCenter.
Remove the decommissioned node's IP address and port information from /usr/local/storediq/etc/siq-elasticsearch.yml on the data servers that use this Elasticsearch cluster.
Enter the command to restart services on the data servers that are using this Elasticsearch cluster:
```
service deepfiler restart
```

Ensure that the cluster works correctly and all nodes are available (now, the decommissioned node is not listed).

curl -X GET 'http://primary_node_ip:9200'
curl -X GET 'http://primary_node_ip:9200/_cluster/health?pretty=true'
curl -X GET 'http://primary_node_ip:9200/_cluster/state?pretty'