Using Avro serialization

Apache Avro is a data serialization system that provides data serialization and data exchange services.

Using Avro serialization programs can efficiently serialize data into a form that is compact and efficient to transmit and can help message consumers manage changes to the structure of the data sent by a producer. To achieve this Avro uses schemas to define the structure of the data in the message.

The IBM® App Connect Enterprise KafkaProducer node allows a JSON format input message to be serialized into Avro format by using a pre-prepared Avro schema before it is published on the Kafka topic. The KafkaProducer node is configured with the schema identifier which identifies the schema to be used to serialize the message, or alternatively the schema identifier might also be provided in the local environment. For more information, see Using local environment variables with Kafka nodes. To use Avro serialization, the KafkaProducer node must be configured with a Schema Registry policy to identify from where the Avro schema is retrieved.

The IBM App Connect Enterprise KafkaConsumer node and IBM App Connect Enterprise KafkaRead allow Avro format messages that are received from Kafka to be deserialized into a JSON format message before they are propagated to subsequent nodes in the message flow. When configured to process Avro messages, the KafkaConsumer and KafkaRead nodes extract the schema identifier from the received message and then fetch the schema from the location that is defined in the Schema Registry policy. The message is then deserialized into a JSON format message by using the Avro schema.