Increasing Kafka retention period

In Business Automation Insights, Kafka retention period is by default set to 2 hours.

About this task

Important: It is recommended to increase the Kafka retention period to extend the time window during which jobs can be restarted without data loss in the event of failure. The retention period must be sufficient so the Kafka topic retains all pending messages until jobs have been restarted.

Increasing Kafka retention increases disk storage consumption so it must be ensured that Kafka disk capacity is sufficient before making the change. See Capability storage requirements.

Procedure

  1. Download Kafka binary files from the Kafka documentation page Download.
  2. Retrieve the certificate in p12 format following Option 3 in Option 1: Retrieving information for connection to Kafka using commands.
  3. Create a connection.properties file and replace values with information obtained in step 2 in Option 1: Retrieving information for connection to Kafka using commands. 
    security.protocol=SSL
ssl.truststore.location=<PATH TO truststore.p12 FILE>
ssl.truststore.password=<TRUSTSTORE_PASSWORD>
ssl.keystore.location=<PATH TO truststore.p12 FILE>
ssl.keystore.password=<TRUSTSTORE_PASSWORD>
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="icp4ba-kafka-auth-0" password="<KAFKA_PASSWORD>";
    Note: Following steps uses kafka-topics.sh tool to configure topics. See Modifying topics of the Kafka documentation for information on this command.
  4. Replace values in following command and execute it to first describe the topic: 
    bin/kafka-topics.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --describe --topic <KAFKA_TOPIC_NAME>
  5. Replace values in following command and execute it to update the Kafka topic retention: 
    bin/kafka-configs.sh --bootstrap-server <BOOTSTRAP_SERVERS> --command-config <PATH TO connection.properties FILE> --alter --topic <KAFKA_TOPIC_NAME> --add-config retention.ms=<RETENTION>