Increasing Kafka retention period

In Business Automation Insights, Kafka retention period is by default set to 2 hours.

Important: It is recommended to increase the Kafka retention period to extend the time window during which jobs can be restarted without data loss in the event of failure. The retention period must be sufficient so the Kafka topic retains all pending messages until jobs have been restarted.

Increasing Kafka retention increases disk storage consumption, which means you must verify that Kafka has sufficient disk capacity before applying the change. See Storage consideration.

Procedure

  1. Download Kafka binary files from the Kafka documentation page Download External link opens a new window or tab.
  2. Retrieve the certificate in p12 format following Option 3 in Option 1: Retrieving information for connection to Kafka using commands.
  3. Create a connection.properties file and replace values with information obtained in step 2 in Option 1: Retrieving information for connection to Kafka using commands. 
    security.protocol=SSL
ssl.truststore.location=<PATH TO truststore.p12 FILE>
ssl.truststore.password=<TRUSTSTORE_PASSWORD>
ssl.keystore.location=<PATH TO truststore.p12 FILE>
ssl.keystore.password=<TRUSTSTORE_PASSWORD>
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="icp4ba-kafka-auth-0" password="<KAFKA_PASSWORD>";
    Note: Following steps uses kafka-topics.sh tool to configure topics. See Modifying topics External link opens a new window or tab of the Kafka documentation for information on this command.
  4. Replace values in following command and execute it to first describe the topic: 
    bin/kafka-topics.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --describe --topic <KAFKA_TOPIC_NAME>
  5. Replace values in following command and execute it to increase the retention value: 
    bin/kafka-configs.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --alter --topic <KAFKA_TOPIC_NAME> --add-config retention.ms=<RETENTION>