Adding the new node to the existing etcd and calicol network

You must add the new HDP node to the existing etcd and calicol network.

1. From the installer node, run the command:

   sudo docker exec -it etcd sh

2. From the etcd container, run the command, replace <hostname of the new node> with the actual value. For example if there are 12 nodes in the cluster, including the new nodes, the index should be 12.

   etcdctl member add etcd<index> http://<hostname of the new node>:2380

   <index> is the total amount of the nodes.

   For example if there are 12 nodes in the cluster, including the new nodes, the index must be 12.

3. In this step, the example uses a three node cluster plus the one new node and assumes the existing cluster hostnames are host1.ibm.com, host2.ibm.com, and host3.ibm.com. The new node hostname is host4.ibm.com.
   1. Run the command:

      etcdctl member add etcd4 http://host4.ibm.com:2380

   2. Make sure the output is similar to:

      ETCD_NAME="etcd4"
      ETCD_INITIAL_CLUSTER="etcd4=http://host4.ibm.com:2380, etcd3=http://host3.ibm.com:2380,etcd2=http://host2.ibm.com:2380,etcd1=http://host1.ibm.com:2380"

   3. Make a record of the value ETCD_INITIAL_CLUSTER that will be used in next step.
4. On the new node, run this commands to create the new etcd container. Make sure initial-cluster value is the same as the output of step 3. Here is the example:

   index=4

   newhost=host4.ibm.com

   sudo docker run -d -p 2380:2380 -p 2379:2379 --name etcd --restart always --net=host --volume=/data/etcd:/etcd ibmiot.ife.registry:5000/quay.io/coreos/etcd:v3.2.4 /usr/local/bin/etcd --data-dir=/etcd-data --name etcd${index} --listen-peer-urls http://${newhost}:2380 --initial-advertise-peer-urls http://${newhost}:2380 --listen-client-urls http://${newhost}:2379 --advertise-client-urls http://${newhost}:2379 --initial-cluster "etcd4=http://host4.ibm.com:2380, etcd3=http://host3.ibm.com:2380,etcd2=http://host2.ibm.com:2380,etcd1=http://host1.ibm.com:2380" --initial-cluster-state existing --initial-cluster-token my-etcd-token

5. Verify the etcd cluster,
   1. From the new node run the command:

      sudo curl -L http://127.0.0.1:2379/v2/members

      You will see all the members in the cluster.
   2. From the installer node run this command in the etcd container:

      etcdctl cluster-health

      The new etcd member shows healthy as do the other members.
6. Start the calicol container. Run the commands on the new node and replace <newnode_ip> with the actual value.

   sudo docker pull ibmiot.ife.registry:5000/quay.io/calico/node:v2.4.0

   sudo calicoctl node run --ip=<newnode_ip> --node-image=ibmiot.ife.registry:5000/quay.io/calico/node:v2.4.0

7. From an existing node, copy the /usr/local/bin/calicoctl to the new node and put it the /usr/local/bin location.
8. Run the command on the new node.

   sudo chmod a+x /usr/bin/calicoctl

9. Verify the status of the calico nodes.
   1. Run this command on the new node

      sudo calicoctl node status

   2. Run the same command on the install node.

      sudo calicoctl node status

   All status of all nodes should be up.
10. Replace the docker.service file on the new node.
    1. On the new node, make a back-up of the /usr/lib/systemd/system/docker.service file.
    2. From an existing node, copy the /usr/lib/systemd/system/docker.service file and replace the file on the new node.
    3. Edit the new /usr/lib/systemd/system/docker.service file to make sure the hostname in the ExecStart line is the new node hostname. For example:

       ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver devicemapper --insecure-registry ibmiot.ife.registry:5000 --cluster-store etcd://host4.ibm.com:2379 --cluster-advertise ens192:2376 --storage-opt dm.basesize=500G

    4. Restart docker services on the new node with the commands:

       sudo systemctl daemon-reload

       sudo systemctl restart docker

11. Do steps 5 and 9 to verify the etcd cluster and the status of the nodes.