Set up Kafka high availability

Apache Kafka is a horizontally scalable, highly available, fault tolerant, low-latency, high-throughput event streaming platform.

A Kafka cluster can be broken down into two components, a control plane and a data plane. Kafka uses the Apache ZooKeeper cluster to provide most of its control plane functionality. ZooKeeper tracks each broker and provides replicated and consistent storage for the cluster metadata. ZooKeeper also elects one Kafka broker to be the controller. For Kafka cluster to be highly available, you must make sure that both control plane and data plane are highly available.

Set up high availability in ZooKeeper

To have a highly available ZooKeeper, use at least three servers. If you have more than three, it's better to use an odd number. By default, one replica of ZooKeeper is available in Telco Network Cloud Manager - Performance. To achieve high availability, a minimum of three ZooKeeper instances are required. To create the three instances to form a quorum, follow these steps:

  1. Log in to your OpenShift® Container Platform web console on your primary cluster.
  2. Expand Workloads > StatefulSets > zookeeper.
  3. Click the YAML tab.
  4. Define the value for the ZOOKEEPER_INSTANCE_ENTRIES parameter in the env section to 3.
    - name: ZOOKEEPER_INSTANCE_ENTRIES
              value: '3'
  5. Click Save.

    The ZooKeeper Pod is restarted automatically after you update the YAML file.

Then, you must manually scale up the Pods to three. Follow these steps:

  1. Open your cloud platform web console.

    https://<master_node_IP>:<Dashboard_externalPort>

  2. Go to Workloads > Stateful Sets and select the zookeeper Service.
  3. Click the Edit resource (Edit resource icon) icon and define the value for the ZOOKEEPER_INSTANCE_ENTRIES parameter in the env section to 3.
    - name: ZOOKEEPER_INSTANCE_ENTRIES
              value: '3'
  4. Click Save.

    The ZooKeeper Pod is restarted automatically and the Pods are scaled up to three.

Verify that the ZooKeeper created the quorum of three cluster by using the following command:
echo srvr | nc zookeeper-<ordinal-number> 2181
echo srvr | nc zookeeper-1 2181
Commnad output

Set up high availability in Kafka

By default, one replica of Kafka is available in Telco Network Cloud Manager - Performance. You can scale to odd or even number of Kafka instances.

Manually scale up the Kafka Pods to the required number.