Setting up an external load balancer
Learn how to configure an external load balancer for your master or proxy nodes in a high availability environment.
If possible, a highly available external load balancer can be used to spread the traffic among separate master or proxy node instances in the cluster. The external load balancer can either be a DNS URL or an IP address, and can be specified by using
the cluster_lb_address
parameter in the config.yaml
file during the installation of your cluster. The cluster_CA_domain
and any TLS certificates must be configured to be a Canonical Name Record (CNAME) or
a record that points to the external load balancer DNS name or IP address. In addition, all nodes in the cluster must be able to resolve this CNAME for internal communication.
When you use an external load balancer, the master load balancer must monitor the Kubernetes API server port 8001 for health on all master nodes, and the load balancer must be configured to accept connections on the following locations:
- Forward traffic to 8001 (Kubernetes API)
- 8443 (platform UI and authentication service)
- 8500 and 8600 (private registry)
Master and proxy nodes in an IBM® Cloud Private high availability environment uses both ucarp and etcd as a load-balancer. With this setup, the virtual IP address (VIP) binds to one master or one proxy node. When you use an external load balancer, each master node can be in a different subnet if the round-trip network time between the master nodes is less than 33 ms for etcd.
You can use an external load balancer as an alternative or replacement for the VIP.
To enable an external load balancer mode in an IBM Cloud Private high availability environment, you must prepare a load balancer node and install HAProxy. Then, configure the load balancer by setting the cluster_lb_address
and the proxy_lb_address
parameters in the config.yaml
file.
The cluster external load balancer is used to load balance the IBM Cloud Private management services. The proxy external load balancer is used to load balance the IBM Cloud Private workload services.
-
To set up a cluster external load balancer, ensure that the following ports are added to the load balancer node and are open: 8001, 8443, 8500, 8600.
-
To set up a proxy external load balancer, ensure that the following ports are added to the load balancer node and are open: 80 and 443.
-
Set up the load balancer node. This load balancer node must not be shared with other cluster nodes such as master, worker, or proxy nodes. A dedicated node is needed to prevent port conflicts.
-
Install HAproxy on the load balancer node.
For Ubuntu:
apt-get install haproxy
Red Hat Enterprise Linux (RHEL):
yum install haproxy
-
Configure HAproxy. Configure HAproxy in the
/etc/haproxy/haproxy.cfg
file on the load balancer node. For example:# Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # # Global settings global # To view messages in the /var/log/haproxy.log you need to: # # 1) Configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog. # # 2) Configure local2 events to go to the /var/log/haproxy.log # file. A line similar to the following can be added to # /etc/sysconfig/syslog. # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # 3) Turn on stats unix socket stats socket /var/lib/haproxy/stats # Common defaults that all the 'listen' and 'backend' sections # use, if not designated in their block. defaults mode http log global option httplog option dontlognull option http-server-close option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 2m timeout server 2m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend k8s-api bind *:8001 mode tcp option tcplog use_backend k8s-api backend k8s-api mode tcp balance roundrobin server server1 <master_node_1_IP_address>:8001 check server server2 <master_node_2_IP_address>:8001 check server server3 <master_node_3_IP_address>:8001 check frontend dashboard bind *:8443 mode tcp option tcplog use_backend dashboard backend dashboard mode tcp balance roundrobin server server1 <master_node_1_IP_address>:8443 check server server2 <master_node_2_IP_address>:8443 check server server3 <master_node_3_IP_address>:8443 check frontend registry bind *:8500 mode tcp option tcplog use_backend registry frontend image-manager bind *:8600 mode tcp option tcplog use_backend image-manager backend image-manager mode tcp balance roundrobin server server1 <master_node_1_IP_address>:8600 check server server2 <master_node_2_IP_address>:8600 check server server3 <master_node_3_IP_address>:8600 check backend registry mode tcp balance roundrobin server server1 <master_node_1_IP_address>:8500 check server server2 <master_node_2_IP_address>:8500 check server server3 <master_node_3_IP_address>:8500 check frontend proxy-http bind *:80 mode tcp option tcplog use_backend proxy-http backend proxy-http mode tcp balance roundrobin server server1 <proxy_node_1_IP_address>:80 check server server2 <proxy_node_2_IP_address>:80 check server server3 <proxy_node_3_IP_address>:80 check frontend proxy-https bind *:443 mode tcp option tcplog use_backend proxy-https backend proxy-https mode tcp balance roundrobin server server1 <proxy_node_1_IP_address>:443 check server server2 <proxy_node_2_IP_address>:443 check server server3 <proxy_node_3_IP_address>:443 check # OPTIONAL: Enable the following Kubernetes NodePorts for applications that require them: frontend proxy-nodeport bind *:30000-32767 mode tcp option tcplog use_backend proxy-nodeport backend proxy-nodeport mode tcp balance roundrobin server server1 <proxy_node_1_IP_address> server server2 <proxy_node_2_IP_address> server server3 <proxy_node_3_IP_address>_
- To set up a cluster load balancer, replace
<master_node_1_IP_address>
,<master_node_2_IP_address>
, and<master_node_3_IP_address>
with the IP addresses for your HA master nodes. - To set up a proxy load balancer, replace
<proxy_node_1_IP_address>
,<proxy_node_2_IP_address>
, and<proxy_node_3_IP_address>
with the IP addresses for your HA proxy nodes.
- To set up a cluster load balancer, replace
-
Start the
haproxy service
by running the following command on the load balancer node:systemctl start haproxy
-
-
Update the
config.yaml
file. Replace thecluster_lb_address
orproxy_lb_address
parameter with the IP address for your external load balancer node. For example:## External loadbalancer IP or domain ## Or floating IP in OpenStack environment cluster_lb_address: none
## External loadbalancer IP or domain ## Or floating IP in OpenStack environment proxy_lb_address: none