PH25635: ADD SUPPORT FOR AVRO MESSAGES TO APACHE KAFKA ADAPTER

APAR status

• Closed as program error.

Error description

• Description: Apache Kafka adapter can currently process messages
  only as raw byte messages. However, messages in Kafka topics
  are often stored as Avro records and can be retrieved from the
  topic as Avro GenericRecord Java objects. When the adapter is
  used to read messages from such a topic today, the messages are
  returned as raw bytes, and it is not possible to present those
  messages in a format that would allow for parsing them and
  performing data transformation on them.
  
  It should be possible to configure the adapter to convert Apache
  Avro records (GenericRecord objects) to JSON text format, so
  that when the messages are modeled in TX using native JSON
  template schemas, the transformation can be performed on those
  messages.
  
  Similarly, it should be possible to configure the adapter to
  take JSON messages on input, along with Avro schema representing
  the format of Avro records, and automatically convert those JSON
  messages to Avro records (GenericRecord objects) and write them
  to the topic.
  

Local fix

• ITXCQ - ITX00060361
  GM / PM
  Circumvention: None
  

Problem summary

• Users Affected:
  ITX users using Apache Kafka adapter.
  
  Problem Description:
  The Kafka adapter supported producing and consuming Kafka
  messages as raw byte messages only. This prevented the use of
  custom serializers and deserializers, which are Java objects
  implemented in separate JARs, that the user can place and
  configure to run between Kafka client (adapter) and Kafka server
  (broker) and convert message key and value data between raw byte
  and other Java type format. One frequently used such Java format
  is Apache Avro GenericRecord type, used to represent Avro
  records. Serializers and deserializers are available that can
  take GenericRecord object from the client (adapter) and store it
  to Kafka topic, and also present Kafka topic Avro messages to
  the client (adapter) as GenericRecord objects. When such a
  serializer or deserializer was utilized with the adapter, the
  produce and consume operations resulted in ClassCastException
  errors because the adapter was not able to handle objects other
  than raw byte arrays.
  
  Platforms Affected:
  All
  

Problem conclusion

• A range of new adapter commands was added to support handling
  messages in Apache Avro format, as well as String format.h.
  
  Applies to:
  9.0.0.3 10.0.0.0
  
  Fixed in the next service packs and releases.
  
  To obtain the fix for this APAR:
  
  To see if the next service pack or product release is available,
  check the IBM Transformation Extender Release Notes page:
  https://www.ibm.com/support/docview.wss?uid=swg27008337
  
  If the service pack or product release is available, download it
  from Fix Central:
  http://www.ibm.com/support/fixcentral/
  
  If the service pack or product release is not available and you
  require the APAR fix immediately, request a Limited Availability
  Interim Fix (LAIF) by opening a case:
  https://www.ibm.com/mysupport/
  
  Prior to version 9.0.0, IBM Transformation Extender was called
  IBM WebSphere Transformation Extender.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  ITX

• Fixed component ID

  5724Q2300

Applicable component levels