High availability IBM® Cloud Private clusters

You can configure high availability (HA) for IBM Cloud Private master and proxy nodes.

You can configure HA for only the master nodes, only the proxy nodes, or for both types of nodes. To reduce the infrastructure requirements of your cluster, you can assign both master and proxy roles to the HA nodes. For the master nodes, the virtual IP manager must be in the same subnet.

Note: To ensure availability, configure more than one proxy node and 3 or 5 master nodes.

You must set up shared storage across your master nodes. IBM Cloud Private requires shared storage for Docker registry. The storage must be a POSIX-compliant shared file system that is located outside of your IBM Cloud Private cluster. Your master nodes must have read/write access to the file system. The file system must be mounted as local hostPath on your master nodes. The following directory must be mounted on your shared storage:

/var/lib/registry - The /var/lib/registry directory stores images in the private image registry and keeps the images that are synchronized on all master nodes.
/var/lib/icp/audit - The /var/lib/icp/audit directory stores pod usage for license check.

Note: You must set the file parameter to 0755 for the directory.

Requirements for master HA

These requirements are for master HA only. Proxy HA is not affected by the number of master nodes.

For N number of masters in a cluster, the cluster can tolerate up to (N-1)/2 permanent failures. For example, in a cluster that has three masters, if one master fails, then the fault tolerance is as (3-1)/2=1. You must aim for a fault tolerance of one or more.

You must have an odd number of masters in your cluster. Adding extra master nodes provides a higher tolerance for failure. You can review how the fault tolerance in a cluster is affected by the number of the master nodes in Table 1:Fault tolerance in HA clusters.

Table 1. Fault tolerance in HA clusters
Number of the master nodes	Failure tolerance
1	0
3	1
5	2
7	3

Node assignment and communication in HA clusters

In HA IBM® Cloud Private clusters, the virtual IP manager controls master and proxy node assignment.

The virtual IP manager controls which nodes serve master and proxy roles by assigning virtual IP addresses to those nodes. The virtual IP manager in IBM Cloud Private facilitates communication between nodes through the network interface controller (NIC). The virtual IP manager assigns the cluster_vip IP address to an available master node and assigns the proxy_vip IP address to an available proxy node. These nodes act as the leading master and proxy node. The cluster_vip IP address must be on the NIC that you specify in the vip_iface parameter. Similarly, the proxy_vip IP addresses must be on the NIC that you specify in the proxy_vip_iface parameter.

The virtual IP manager monitors the health of the cluster's master and proxy nodes. If the leading master or proxy node is no longer available, the virtual IP manager selects an available node and assigns it to the correct virtual IP address.

While IBM Cloud Private manages HA through the virtual IP manager, you can also use an external load balancer to distribute the load of the master and proxy nodes and facilitate external communication. To use a load balancer, during installation, specify its IP address as the cluster_lb_address and proxy_lb_address parameter values in the config.yaml file.

Note: For a HA environment, you must set up at least one of the following parameters: cluster_vip, cluster_lb_address.

For more details on setting up HA during installation, see HA installation options.