Setting up an external load balancer

Learn how to configure an external load balancer for your master or proxy nodes in a high availability environment.

If possible, a highly available external load balancer can be used to spread the traffic among separate master or proxy node instances in the cluster. The external load balancer can either be a DNS URL or an IP address, and can be specified by using the cluster_lb_address parameter in the config.yaml file during the installation of your cluster. The cluster_CA_domain and any TLS certificates must be configured to be a Canonical Name Record (CNAME) or a record that points to the external load balancer DNS name or IP address. In addition, all nodes in the cluster must be able to resolve this CNAME for internal communication.

When you use an external load balancer, the master load balancer must monitor the Kubernetes API server port 8001 for health on all master nodes, and the load balancer must be configured to accept connections on the following locations:

Master and proxy nodes in an IBM® Cloud Private high availability environment uses both ucarp and etcd as a load-balancer. With this setup, the virtual IP address (VIP) binds to one master or one proxy node. When you use an external load balancer, each master node can be in a different subnet if the round-trip network time between the master nodes is less than 33 ms for etcd.

You can use an external load balancer as an alternative or replacement for the VIP.

To enable an external load balancer mode in an IBM Cloud Private high availability environment, you must prepare a load balancer node and install HAProxy. Then, configure the load balancer by setting the cluster_lb_address and the proxy_lb_address parameters in the config.yaml file.

The cluster external load balancer is used to load balance the IBM Cloud Private management services. The proxy external load balancer is used to load balance the IBM Cloud Private workload services.

  1. To set up a cluster external load balancer, ensure that the following ports are added to the load balancer node and are open: 8001, 8443, 8500, 8600.

  2. To set up a proxy external load balancer, ensure that the following ports are added to the load balancer node and are open: 80 and 443.

  3. Set up the load balancer node. This load balancer node must not be shared with other cluster nodes such as master, worker, or proxy nodes. A dedicated node is needed to prevent port conflicts.

    1. Install HAproxy on the load balancer node.

      For Ubuntu:

      apt-get install haproxy
      

      Red Hat Enterprise Linux (RHEL):

      yum install haproxy
      
    2. Configure HAproxy. Configure HAproxy in the /etc/haproxy/haproxy.cfg file on the load balancer node. For example:

        # Example configuration for a possible web application.  See the
        # full configuration options online.
        #
        #   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
        #
        # Global settings
        global
        # To view messages in the /var/log/haproxy.log you need to:
        #
        # 1) Configure syslog to accept network log events.  This is done
        #    by adding the '-r' option to the SYSLOGD_OPTIONS in
        #    /etc/sysconfig/syslog.
        #
        # 2) Configure local2 events to go to the /var/log/haproxy.log
        #   file. A line similar to the following can be added to
        #   /etc/sysconfig/syslog.
        #
        #    local2.*                       /var/log/haproxy.log
        #
        log         127.0.0.1 local2
      
        chroot      /var/lib/haproxy
        pidfile     /var/run/haproxy.pid
        maxconn     4000
        user        haproxy
        group       haproxy
        daemon
      
        # 3) Turn on stats unix socket
        stats socket /var/lib/haproxy/stats
        # Common defaults that all the 'listen' and 'backend' sections
        # use, if not designated in their block.
        defaults
        mode                    http
        log                     global
        option                  httplog
        option                  dontlognull
        option http-server-close
        option                  redispatch
        retries                 3
        timeout http-request    10s
        timeout queue           1m
        timeout connect         10s
        timeout client          2m
        timeout server          2m
        timeout http-keep-alive 10s
        timeout check           10s
        maxconn                 3000
      
        frontend k8s-api
        bind *:8001
        mode tcp
        option tcplog
        use_backend k8s-api
      
        backend k8s-api
        mode tcp
        balance roundrobin
        server server1 <master_node_1_IP_address>:8001 check
        server server2 <master_node_2_IP_address>:8001 check
        server server3 <master_node_3_IP_address>:8001 check
      
        frontend dashboard
        bind *:8443
        mode tcp
        option tcplog
        use_backend dashboard
      
        backend dashboard
        mode tcp
        balance roundrobin
        server server1 <master_node_1_IP_address>:8443 check
        server server2 <master_node_2_IP_address>:8443 check
        server server3 <master_node_3_IP_address>:8443 check
      
        frontend registry
        bind *:8500
        mode tcp
        option tcplog
        use_backend registry
      
        frontend image-manager
        bind *:8600
        mode tcp
        option tcplog
        use_backend image-manager
      
        backend image-manager
        mode tcp
        balance roundrobin
        server server1 <master_node_1_IP_address>:8600 check
        server server2 <master_node_2_IP_address>:8600 check
        server server3 <master_node_3_IP_address>:8600 check
      
        backend registry
        mode tcp
        balance roundrobin
        server server1 <master_node_1_IP_address>:8500 check
        server server2 <master_node_2_IP_address>:8500 check
        server server3 <master_node_3_IP_address>:8500 check
      
        frontend proxy-http
        bind *:80
        mode tcp
        option tcplog
        use_backend proxy-http
      
        backend proxy-http
        mode tcp
        balance roundrobin
        server server1 <proxy_node_1_IP_address>:80 check
        server server2 <proxy_node_2_IP_address>:80 check
        server server3 <proxy_node_3_IP_address>:80 check
      
        frontend proxy-https
        bind *:443
        mode tcp
        option tcplog
        use_backend proxy-https
      
        backend proxy-https
        mode tcp
        balance roundrobin
        server server1 <proxy_node_1_IP_address>:443 check
        server server2 <proxy_node_2_IP_address>:443 check
        server server3 <proxy_node_3_IP_address>:443 check
      
        # OPTIONAL: Enable the following Kubernetes NodePorts for applications that require them:
        frontend proxy-nodeport
        bind *:30000-32767
        mode tcp
        option tcplog
        use_backend proxy-nodeport
      
        backend proxy-nodeport
        mode tcp
        balance roundrobin
        server server1 <proxy_node_1_IP_address>
        server server2 <proxy_node_2_IP_address>
        server server3 <proxy_node_3_IP_address>_
      
      • To set up a cluster load balancer, replace <master_node_1_IP_address>, <master_node_2_IP_address>, and <master_node_3_IP_address> with the IP addresses for your HA master nodes.
      • To set up a proxy load balancer, replace <proxy_node_1_IP_address>, <proxy_node_2_IP_address>, and <proxy_node_3_IP_address> with the IP addresses for your HA proxy nodes.
    3. Start the haproxy service by running the following command on the load balancer node:

        systemctl start haproxy
      
  4. Update the config.yaml file. Replace the cluster_lb_address or proxy_lb_address parameter with the IP address for your external load balancer node. For example:

     ## External loadbalancer IP or domain
     ## Or floating IP in OpenStack environment
     cluster_lb_address: none
    
     ## External loadbalancer IP or domain
     ## Or floating IP in OpenStack environment
     proxy_lb_address: none