Increasing Kafka partitions

To increase the number of partitions in a Kafka topic, follow these steps to configure the connection, describe the topic, and apply the necessary changes.

About this task

See Kafka official documentation External link opens a new window or tab

in order to increase the number of Kafka topic partitions

Procedure

Download Kafka binary files from the Kafka documentation page Download .
Retrieve the certificate in p12 format following Option 3 in Option 1: Retrieving information for connection to Kafka using commands.

Create a connection.properties file and replace values with information obtained in step 2 in Retrieving information for connection to Kafka using commands.

security.protocol=SSL
ssl.truststore.location=<PATH TO truststore.p12 FILE>
ssl.truststore.password=<TRUSTSTORE_PASSWORD>
ssl.keystore.location=<PATH TO truststore.p12 FILE>
ssl.keystore.password=<TRUSTSTORE_PASSWORD>
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="icp4ba-kafka-auth-0" password="<KAFKA_PASSWORD>";

Note: Following steps uses kafka-topics.sh tool to configure topics. See Modifying topics External link opens a new window or tab

of the Kafka documentation for information on this command.

Replace values in following command and execute it to first describe the topic:

bin/kafka-topics.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --describe --topic <KAFKA_TOPIC_NAME>

Replace values in following command and execute it to increase the number of partitions:

bin/kafka-topics.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --alter --topic <KAFKA_TOPIC_NAME> --partitions <NUMBER_OF_PARTITIONS>

After the topic configuration is adjusted, stop the Flink jobs and then restart them.

oc get job <job-name> -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | oc replace --force -f -