KafkaProducer node

Use the KafkaProducer node to connect to the Apache Kafka messaging system, and to publish messages from a message flow to a topic on a Kafka server.

Purpose

You can use a KafkaProducer node in a message flow to publish an output message from a message flow to a specified topic on a Kafka server. You set properties on the KafkaProducer node to define how it will connect to the Kafka messaging system, and to specify the topic to which messages are sent.

For more information, see Processing Kafka messages.

The KafkaProducer node handles messages in the following message domains:

  • DFDL
  • XMLNSC
  • JSON
  • BLOB
  • MIME
  • XMLNS
  • MRM

The KafkaProducer node is contained in the Kafka drawer of the palette, and is represented in the IBM App Connect Enterprise Toolkit by the following icon:

KafkaProducer node icon

Using the KafkaProducer node in a message flow

Use the KafkaProducer node to publish messages that are generated from within your message flow to a topic that is hosted on a Kafka server. The published messages are then delivered by the Kafka server to all topic subscribers (consumers).

The KafkaProducer node publishes messages non-transactionally to the Kafka server, and they are available to be read by consuming applications as soon as they are published. As the publish operation is non-transactional, if the flow is rolled back to the input node after the message has passed through the KafkaProducer node, the publication of the message to the Kafka server is not rolled back.

You can use the Acks property on the KafkaProducer node to configure synchronous processing of the message, by specifying that the KafkaProducer node must wait for confirmation that the message has been successfully received by the Kafka server before continuing in the flow.

For more information about how to use the node, see Producing messages on Kafka topics.

For information about the supported versions of Kafka, see IBM App Connect Enterprise system requirements. For more information about Kafka version compatibility, see the Apache Kafka documentation.

Terminals and properties

The KafkaProducer node terminals are described in the following table.

Terminal Description
In The input terminal that accepts a message for processing by the node.
Failure The output terminal to which the message is routed if a failure is detected during processing in the node.
Out The output terminal to which the message is routed if it is successfully published to the Kafka topic.

The following tables describe the node properties. The column headed M indicates whether the property is mandatory (marked with an asterisk on the panel if you must enter a value when no default is defined); the column headed C indicates whether the property is configurable (you can change the value when you add the message flow to the BAR file to deploy it).

The KafkaProducer node Description properties are described in the following table.

Property M C Default Description
Node name No No The node type, KafkaProducer The name of the node.
Short description No No   A brief description of the node.
Long description No No   Text that describes the purpose of the node in the message flow.

The KafkaProducer node Basic properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Topic name Yes Yes   The name of the Kafka topic where the message will be published. Only one topic name can be specified. The name can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). topicName
Bootstrap servers Yes Yes   A list of host/port pairs (separated by commas) to use for establishing the initial connection to the Kafka cluster. bootstrapServers
Client ID No Yes   The client name to be used when connecting to Kafka. This value can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). clientId
Add server name suffix to client ID No Yes Yes If selected, this property suffixes the provided client ID with either -integrationServerName (for an independent integration server) or -integrationNodeName-integrationServerName (for an integration server that is managed by an integration node). This option is selected by default. useClientIdSuffix
Acks No Yes 0 The number of acks (acknowledgements) to request from the server before the publication request is sent. Possible values are:
  • 0
  • 1
  • All

If this property is set to 0, the KafkaProducer node does not wait for any acknowledgement that the publish request has been processed by the Kafka server. This is equivalent to a 'fire and forget' mode of operation.

If this property is set to 1, the KafkaProducer node waits for a single acknowledgement from the Kafka server.

If this property is set to All, the KafkaProducer node waits for acknowledgements from all replicas of the topic. This option provides the strongest available guarantee that the record was received.

acks
Timeout (sec) No Yes 60 The time (in seconds) to wait for the request to complete. timeout

The KafkaProducer node Advanced properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Properties file No Yes null Full path name of a kafka.properties file to set Kafka client properties.

You can use a properties file to set properties on the Kafka producer that cannot be set through the KafkaProducer node properties. If the properties file specifies a property that is also set on the KafkaProducer node, the value set in the properties file takes precedence. For information about the complete list of properties that are available, see the Apache Kafka documentation.

propertiesFilePath

The KafkaProducer node Security properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Security identifier No Yes null Identifier that is used to look up the user ID and password from the App Connect Enterprise vault, which was defined by using the mqsicredentials command.

If this property is left blank, the default kafka credential (defined by using the mqsicredentials command) is used. If a default credential has not been defined, the security ID KAFKA::KAFKA::{integrationServerName} is used.

securityIdentity
Security protocol Yes Yes PLAINTEXT The protocol to be used when communicating with integration nodes. Valid values are:
  • PLAINTEXT
  • SSL
  • SASL_PLAINTEXT
  • SASL_SSL

If you are using IBM Event Streams, this property must be set to SASL_SSL.

If either SASL_PLAINTEXT or SASL_SSL is selected, you must configure the user ID and password that will be used to authenticate with the Kafka server. You can configure these credentials by using the mqsicredentials command (or the mqsisetdbparms command). For more information, see Configuring security credentials for connecting to Kafka.

securityProtocol
SSL protocol Yes Yes TLSv1_2 The SSL protocol that is used when the selected value for the Security protocol property is either SSL or SASL_SSL. You can select one of the following values from the editable list, or you can specify an alternative value:
  • TLSv1
  • TLSv1.1
  • TLSv1.2
sslProtocol

The KafkaProducer node Policy properties are described in the following table.

Property M C Default Description mqsiapplybaroverride command property
Policy No Yes null Name of a Kafka policy to be used to override node properties; for example, {PolicyProject}:kafkaPolicy policyUrl

The KafkaProducer node Validation properties are described in the following table.

Property M Default Description
Validate No Inherit This property controls whether validation takes place. Valid values are None, Content and Value, Content, and Inherit.
Failure action No Exception This property controls what happens if validation fails. You can set this property only if you set Validate to Content or Content and Value. Valid values are User Trace, Local Error Log, Exception, and Exception List.

For a full description of these properties, see Validation properties.

The Monitoring properties of the node are described in the following table.
Property M C Default Description
Events No No None Events that you have defined for the node are displayed on this tab. By default, no monitoring events are defined on any node in a message flow. Use Add, Edit, and Delete to create, change or delete monitoring events for the node.

You can enable and disable events that are shown here by selecting or clearing the Enabled check box.