Adding the new node to the existing etcd and calicol network

You must add the new HDP node to the existing etcd and calicol network.


  1. From the installer node, run the command:
    sudo docker exec -it etcd sh
  2. From the etcd container, run the command, replace <hostname of the new node> with the actual value. For example if there are 12 nodes in the cluster, including the new nodes, the index should be 12.
    etcdctl member add etcd<index> http://<hostname of the new node>:2380
    <index> is the total amount of the nodes.

    For example if there are 12 nodes in the cluster, including the new nodes, the index must be 12.

  3. In this step, the example uses a three node cluster plus the one new node and assumes the existing cluster hostnames are,, and The new node hostname is
    1. Run the command:
      etcdctl member add etcd4
    2. Make sure the output is similar to:
      ETCD_INITIAL_CLUSTER="etcd4=, etcd3=,etcd2=,etcd1="
    3. Make a record of the value ETCD_INITIAL_CLUSTER that will be used in next step.
  4. On the new node, run this commands to create the new etcd container. Make sure initial-cluster value is the same as the output of step 3. Here is the example:
    sudo docker run -d -p 2380:2380 -p 2379:2379 --name etcd --restart always --net=host --volume=/data/etcd:/etcd ibmiot.ife.registry:5000/ /usr/local/bin/etcd --data-dir=/etcd-data --name etcd${index} --listen-peer-urls http://${newhost}:2380 --initial-advertise-peer-urls http://${newhost}:2380 --listen-client-urls http://${newhost}:2379 --advertise-client-urls http://${newhost}:2379 --initial-cluster "etcd4=, etcd3=,etcd2=,etcd1=" --initial-cluster-state existing --initial-cluster-token my-etcd-token
  5. Verify the etcd cluster,
    1. From the new node run the command:
      sudo curl -L
      You will see all the members in the cluster.
    2. From the installer node run this command in the etcd container:
      etcdctl cluster-health
      The new etcd member shows healthy as do the other members.
  6. Start the calicol container. Run the commands on the new node and replace <newnode_ip> with the actual value.
    sudo docker pull ibmiot.ife.registry:5000/
    sudo calicoctl node run --ip=<newnode_ip> --node-image=ibmiot.ife.registry:5000/
  7. From an existing node, copy the /usr/local/bin/calicoctl to the new node and put it the /usr/local/bin location.
  8. Run the command on the new node.
    sudo chmod a+x /usr/bin/calicoctl
  9. Verify the status of the calico nodes.
    1. Run this command on the new node
      sudo calicoctl node status
    2. Run the same command on the install node.
      sudo calicoctl node status
    All status of all nodes should be up.
  10. Replace the docker.service file on the new node.
    1. On the new node, make a back-up of the /usr/lib/systemd/system/docker.service file.
    2. From an existing node, copy the /usr/lib/systemd/system/docker.service file and replace the file on the new node.
    3. Edit the new /usr/lib/systemd/system/docker.service file to make sure the hostname in the ExecStart line is the new node hostname. For example:
      ExecStart=/usr/bin/dockerd -H tcp:// -H unix:///var/run/docker.sock --storage-driver devicemapper --insecure-registry ibmiot.ife.registry:5000 --cluster-store etcd:// --cluster-advertise ens192:2376 --storage-opt dm.basesize=500G
    4. Restart docker services on the new node with the commands:
      sudo systemctl daemon-reload
      sudo systemctl restart docker
  11. Do steps 5 and 9 to verify the etcd cluster and the status of the nodes.