Increasing Kafka retention period

Kafka retention period is by default set to 2 hours in IBM for Cloud Pak Business Automation. Follow this procedure to update the Kafka topic retention and make events persist in Kafka for a longer period of time.

Procedure

  1. Download Kafka binary files from the Kafka documentation page Download.
  2. Retrieve the certificate in p12 format following Option 3 in Option 1: Retrieving information for connection to Kafka using commands.
  3. Create a connection.properties file and replace values with information obtained in step 2 in Option 1: Retrieving information for connection to Kafka using commands. 
    security.protocol=SSL
ssl.truststore.location=<PATH TO truststore.p12 FILE>
ssl.truststore.password=<TRUSTSTORE_PASSWORD>
ssl.keystore.location=<PATH TO truststore.p12 FILE>
ssl.keystore.password=<TRUSTSTORE_PASSWORD>
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="icp4ba-kafka-auth-0" password="<KAFKA_PASSWORD>";
    Note: Following steps uses kafka-topics.sh tool to configure topics. See Modifying topics of the Kafka documentation for information on this command.
  4. Replace values in following command and execute it to first describe the topic: 
    bin/kafka-topics.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --describe --topic <KAFKA_TOPIC_NAME>
  5. Replace values in following command and execute it to update the Kafka topic retention: 
    bin/kafka-configs.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --alter --topic <KAFKA_TOPIC_NAME> --add-config retention.ms=<RETENTION>