Setting up an external load balancer

Learn how to configure an external load balancer for your master or proxy nodes in a high availability environment.

If possible, a highly available external load balancer can be used to spread the traffic among separate master or proxy node instances in the cluster. The external load balancer can either be a DNS URL or an IP address, and can be specified by using the cluster_lb_address parameter in the config.yaml file during the installation of your cluster. The cluster_CA_domain and any TLS certificates must be configured to be a Canonical Name Record (CNAME) or a record that points to the external load balancer DNS name or IP address. In addition, all nodes in the cluster must be able to resolve this CNAME for internal communication.

When you use an external load balancer, the master load balancer must monitor the Kubernetes API server port 8001 for health on all master nodes, and the load balancer must be configured to accept connections on the following locations:

Forward traffic to 8001 (Kubernetes API)
8443 (platform UI and authentication service)
8500 and 8600 (private registry)

Master and proxy nodes in an IBM® Cloud Private high availability environment uses both ucarp and etcd as a load-balancer. With this setup, the virtual IP address (VIP) binds to one master or one proxy node. When you use an external load balancer, each master node can be in a different subnet if the round-trip network time between the master nodes is less than 33 ms for etcd.

You can use an external load balancer as an alternative or replacement for the VIP.

To enable an external load balancer mode in an IBM Cloud Private high availability environment, you must prepare a load balancer node and install HAProxy. Then, configure the load balancer by setting the cluster_lb_address and the proxy_lb_address parameters in the config.yaml file.

The cluster external load balancer is used to load balance the IBM Cloud Private management services. The proxy external load balancer is used to load balance the IBM Cloud Private workload services.

To set up a cluster external load balancer, ensure that the following ports are added to the load balancer node and are open: 8001, 8443, 8500, 8600, 9443.
To set up a proxy external load balancer, ensure that the following ports are added to the load balancer node and are open: 80 and 443.

Set up the load balancer node. This load balancer node must not be shared with other cluster nodes such as master, worker, or proxy nodes. A dedicated node is needed to prevent port conflicts.

Install HAproxy on the load balancer node.

For Ubuntu:
```
 apt-get install haproxy
```
Red Hat Enterprise Linux (RHEL):
```
 yum install haproxy
```

Configure HAproxy. Configure HAproxy in the /etc/haproxy/haproxy.cfg file on the load balancer node. For example:

 # Example configuration for a possible web application.  See the
 # full configuration options online.
 #
 #   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
 #
 # Global settings
 global
 # To view messages in the /var/log/haproxy.log you need to:
 #
 # 1) Configure syslog to accept network log events.  This is done
 #    by adding the '-r' option to the SYSLOGD_OPTIONS in
 #    /etc/sysconfig/syslog.
 #
 # 2) Configure local2 events to go to the /var/log/haproxy.log
 #   file. A line similar to the following can be added to
 #   /etc/sysconfig/syslog.
 #
 #    local2.*                       /var/log/haproxy.log
 #
 log         127.0.0.1 local2

 chroot      /var/lib/haproxy
 pidfile     /var/run/haproxy.pid
 maxconn     4000
 user        haproxy
 group       haproxy
 daemon

 # 3) Turn on stats unix socket
 stats socket /var/lib/haproxy/stats
 # Common defaults that all the 'listen' and 'backend' sections
 # use, if not designated in their block.
 defaults
 mode                    http
 log                     global
 option                  httplog
 option                  dontlognull
 option http-server-close
 option                  redispatch
 retries                 3
 timeout http-request    10s
 timeout queue           1m
 timeout connect         10s
 timeout client          1m
 timeout server          1m
 timeout http-keep-alive 10s
 timeout check           10s
 maxconn                 3000

 frontend k8s-api
 bind *:8001
 mode tcp
 option tcplog
 use_backend k8s-api

 backend k8s-api
 mode tcp
 balance roundrobin
 server server1 <master_node_1_IP_address>:8001 check
 server server2 <master_node_2_IP_address>:8001 check
 server server3 <master_node_3_IP_address>:8001 check

 frontend dashboard
 bind *:8443
 mode tcp
 option tcplog
 use_backend dashboard

 backend dashboard
 mode tcp
 balance roundrobin
 server server1 <master_node_1_IP_address>:8443 check
 server server2 <master_node_2_IP_address>:8443 check
 server server3 <master_node_3_IP_address>:8443 check

 frontend auth
 bind *:9443
 mode tcp
 option tcplog
 use_backend auth

 backend auth
 mode tcp
 balance roundrobin
 server server1 <master_node_1_IP_address>:9443 check
 server server2 <master_node_2_IP_address>:9443 check
 server server3 <master_node_3_IP_address>:9443 check

 frontend registry
 bind *:8500
 mode tcp
 option tcplog
 use_backend registry

 frontend image-manager
 bind *:8600
 mode tcp
 option tcplog
 use_backend image-manager

 backend image-manager
 mode tcp
 balance roundrobin
 server server1 <master_node_1_IP_address>:8600 check
 server server2 <master_node_2_IP_address>:8600 check
 server server3 <master_node_3_IP_address>:8600 check

 backend registry
 mode tcp
 balance roundrobin
 server server1 <master_node_1_IP_address>:8500 check
 server server2 <master_node_2_IP_address>:8500 check
 server server3 <master_node_3_IP_address>:8500 check

 frontend proxy-http
 bind *:80
 mode tcp
 option tcplog
 use_backend proxy-http

 backend proxy-http
 mode tcp
 balance roundrobin
 server server1 <proxy_node_1_IP_address>:80 check
 server server2 <proxy_node_2_IP_address>:80 check
 server server3 <proxy_node_3_IP_address>:80 check

 frontend proxy-https
 bind *:443
 mode tcp
 option tcplog
 use_backend proxy-https

 backend proxy-https
 mode tcp
 balance roundrobin
 server server1 <proxy_node_1_IP_address>:443 check
 server server2 <proxy_node_2_IP_address>:443 check
 server server3 <proxy_node_3_IP_address>:443 check

 # OPTIONAL: Enable the following Kubernetes NodePorts for applications that require them:
 frontend proxy-nodeport
 bind *:30000-32767
 mode tcp
 option tcplog
 use_backend proxy-nodeport

 backend proxy-nodeport
 mode tcp
 balance roundrobin
 server server1 <proxy_node_1_IP_address>
 server server2 <proxy_node_2_IP_address>
 server server3 <proxy_node_3_IP_address>_

To set up a cluster load balancer, replace <master_node_1_IP_address>, <master_node_2_IP_address>, and <master_node_3_IP_address> with the IP addresses for your HA master nodes.
To set up a proxy load balancer, replace <proxy_node_1_IP_address>, <proxy_node_2_IP_address>, and <proxy_node_3_IP_address> with the IP addresses for your HA proxy nodes.

Start the haproxy service by running the following command on the load balancer node:
```
systemctl start haproxy
```

Update the config.yaml file. Replace the cluster_lb_address or proxy_lb_address parameter with the IP address for your external load balancer node. For example:

 ## External loadbalancer IP or domain
 ## Or floating IP in OpenStack environment
 cluster_lb_address: none

 ## External loadbalancer IP or domain
 ## Or floating IP in OpenStack environment
 proxy_lb_address: none