Setting and retrieving Kafka custom header properties

You can use Kafka custom header properties to associate metadata with a Kafka message.

Before you begin

Read the following topics:

About this task

Kafka custom header properties enable you to add metadata to the Kafka message, which can then be used during message processing. For example, the header properties can carry information about the format of the data, such as a schema name.

Kakfa custom header properties are a set of name:value pairs, which can be associated with a Kafka message when it is published and then retrieved when the message is retrieved. IBM App Connect Enterprise supports Kafka header properties that consist of a name in Java String format and a value.

The properties are set in the Local Environment, in a folder called KafkaHeader. For the KafkaProducer node, all property values except for BLOB values are transformed into UTF-8 strings before the bytes are stored in the header property in the Kafka message. For BLOB values, the bytes are stored in the header property of the Kafka message without being transformed. For KafkaConsumer and KafkaRead nodes, the values of header properties in the received message are converted to string values when they are written into the Local Environment, by default. To receive the values as bytes instead of strings, set the Header scheme property on the node to Bytes.

Before calling the KafkaProducer node, you can set the Kafka custom header properties in the Environment.Destination.Kafka.Output.KafkaHeader Local Environment folder. For example:

# To set Kafka custom header properties called 'Name' and 'Occupation'
      SET OutputLocalEnvironment.Destination.Kafka.Output.KafkaHeader.Name = 'Bob';
      SET OutputLocalEnvironment.Destination.Kafka.Output.KafkaHeader.Occupation = 'Builder';

On return from a KafkaConsumer node, the Local Environment folder LocalEnvironment.Kafka.Input is updated with any custom headers properties contained in the received message, which can then be processed by using a Compute node. For example:

 # To retrieve the value of the custom header property called 'Name'
        SET Name = InputLocalEnvironment.Kafka.Input.KafkaHeader.Name;

On return from a KafkaRead node, the Local Environment folder LocalEnvironment.Kafka.Read is updated with any custom headers properties contained in the received message, which can then be processed by using a Compute node. For example:

# To retrieve the value of the custom header property called 'Occupation'
        SET Occupation = InputLocalEnvironment.Kafka.Read.KafkaHeader.Occupation;

For more information about the Kafka nodes, see the following topics:

For information about properties that can be overridden dynamically in the flow, see Using local environment variables with Kafka nodes.