KAFKA_41 - Timeout expired while fetching topic metadata

Problem

Let's say you have 3 brokers in the bootstrap list and the first broker is not reachable and the request is made from the client to the first broker. This request will get a time-out because the first broker is not reachable and the pipeline will not make a retry to the next available broker, instead, it will fail with an above “timeout” error.

In an ideal scenario, the client/pipeline should traverse through all the brokers in the list before marking it as a failure.

server1:9092,server2:9092,server3:9092

This is a known issue in kafka library v2.6 and below which is external to the SDC.

Solution

There are two solutions to this problem:

• Solution 1 (Recommended)

  1. Upgrade the Kafka client library in the stage to 2.7 or above and tune socket timeouts accordingly. In this version, Kafka introduced two new configurations (see below) that make socket timeout to be controlled on the client side.

socket.connection.setup.timeout.max.ms

socket.connection.setup.timeout.ms

b. To upgrade the Kafka library to 2.7 or above, you can go to Kafka consumer origin or Kafka producer stage > Configuration > General > and choose the appropriate Stage Library above 2.7 as shown in the below image.

• Solution 2

  1. Decrease the TCP retry value from the sdc running server-side file  /proc/sys/net/ipv4/tcp_syn_retries to 3. (The default is 6)