KafkaProducer node
Use the KafkaProducer node to connect to the Apache Kafka messaging system, and to publish messages from a message flow to a topic on a Kafka server.
This topic contains the following sections:
Purpose
You can use a KafkaProducer node in a message flow to publish an output message from a message flow to a specified topic on a Kafka server. You set properties on the KafkaProducer node to define how it will connect to the Kafka messaging system, and to specify the topic to which messages are sent.
For more information, see Processing Kafka messages.
The KafkaProducer node handles messages in the following message domains:
- DFDL
- XMLNSC
- JSON
- BLOB
- MIME
- XMLNS
- MRM
The KafkaProducer node is contained in the Kafka drawer of the palette, and is represented in the IBM App Connect Enterprise Toolkit by the following icon:
Using the KafkaProducer node in a message flow
Use the KafkaProducer node to publish messages that are generated from within your message flow to a topic that is hosted on a Kafka server. The published messages are then delivered by the Kafka server to all topic subscribers (consumers).
The KafkaProducer node publishes messages non-transactionally to the Kafka server, and they are available to be read by consuming applications as soon as they are published. As the publish operation is non-transactional, if the flow is rolled back to the input node after the message has passed through the KafkaProducer node, the publication of the message to the Kafka server is not rolled back.
You can use the Acks property on the KafkaProducer node to configure synchronous processing of the message, by specifying that the KafkaProducer node must wait for confirmation that the message has been successfully received by the Kafka server before continuing in the flow.
For more information about how to use the node, see Producing messages on Kafka topics.
For information about the supported versions of Kafka, see IBM App Connect Enterprise system requirements. For more information about Kafka version compatibility, see the Apache Kafka documentation.
Terminals and properties
The KafkaProducer node terminals are described in the following table.
Terminal | Description |
---|---|
In | The input terminal that accepts a message for processing by the node. |
Failure | The output terminal to which the message is routed if a failure is detected during processing in the node. |
Out | The output terminal to which the message is routed if it is successfully published to the Kafka topic. |
The following tables describe the node properties. The column headed M indicates whether the property is mandatory (marked with an asterisk on the panel if you must enter a value when no default is defined); the column headed C indicates whether the property is configurable (you can change the value when you add the message flow to the BAR file to deploy it).
The KafkaProducer node Description properties are described in the following table.
Property | M | C | Default | Description |
---|---|---|---|---|
Node name | No | No | The node type, KafkaProducer | The name of the node. |
Short description | No | No | A brief description of the node. | |
Long description | No | No | Text that describes the purpose of the node in the message flow. |
The KafkaProducer node Basic properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Topic name | Yes | Yes | The name of the Kafka topic where the message will be published. Only one topic name can be specified. The name can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). | topicName | |
Bootstrap servers | Yes | Yes | A list of host/port pairs (separated by commas) to use for establishing the initial connection to the Kafka cluster. | bootstrapServers | |
Client ID | No | Yes | The client name to be used when connecting to Kafka. This value can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). | clientId | |
Add server name suffix to client ID | No | Yes | Yes | If selected, this property suffixes the provided client ID with
either -integrationServerName (for an independent integration
server) or -integrationNodeName-integrationServerName (for an
integration server that is managed by an integration node). This option is selected by
default. |
useClientIdSuffix |
Acks | No | Yes | 0 | The number of acks (acknowledgements) to request from the
server before the publication request is sent. Possible values are:
If this property is set to 0, the KafkaProducer node does not wait for any acknowledgement that the publish request has been processed by the Kafka server. This is equivalent to a 'fire and forget' mode of operation. If this property is set to 1, the KafkaProducer node waits for a single acknowledgement from the Kafka server. If this property is set to All, the KafkaProducer node waits for acknowledgements from all replicas of the topic. This option provides the strongest available guarantee that the record was received. |
acks |
Timeout (sec) | No | Yes | 60 | The time (in seconds) to wait for the request to complete. | timeout |
The KafkaProducer node Advanced properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Properties file | No | Yes | null | Full path name of a kafka.properties file to set Kafka
client properties. You can use a properties file to set properties on the Kafka producer that cannot be set through the KafkaProducer node properties. If the properties file specifies a property that is also set on the KafkaProducer node, the value set in the properties file takes precedence. For information about the complete list of properties that are available, see the Apache Kafka documentation. |
propertiesFilePath |
The KafkaProducer node Security properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Security identifier | No | Yes | null | Identifier that is used to look up the user ID and password from the App Connect Enterprise vault, which was defined by using the mqsicredentials command. If this property is left blank, the default
|
securityIdentity |
Security protocol | Yes | Yes | PLAINTEXT | The protocol to be used when communicating with integration nodes. Valid
values are:
If you are using IBM Event Streams, this property must be set to SASL_SSL. If either SASL_PLAINTEXT or SASL_SSL is selected, you must configure the user ID and password that will be used to authenticate with the Kafka server. You can configure these credentials by using the mqsicredentials command (or the mqsisetdbparms command). For more information, see Configuring security credentials for connecting to Kafka. |
securityProtocol |
SSL protocol | Yes | Yes | TLSv1_2 | The SSL protocol that is used when the selected value for the Security protocol property is either SSL or SASL_SSL.
You can select one of the following values from the editable list, or you can specify an alternative
value:
|
sslProtocol |
The KafkaProducer node Policy properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Policy | No | Yes | null | Name of a Kafka policy to be used to override node properties; for example,
{PolicyProject}:kafkaPolicy
|
policyUrl |
The KafkaProducer node Validation properties are described in the following table.
Property | M | Default | Description |
---|---|---|---|
Validate | No | Inherit | This property controls whether validation takes place. Valid values are None, Content and Value, Content, and Inherit. |
Failure action | No | Exception | This property controls what happens if validation fails. You can set this property only if you set Validate to Content or Content and Value. Valid values are User Trace, Local Error Log, Exception, and Exception List. |
For a full description of these properties, see Validation properties.
Property | M | C | Default | Description |
---|---|---|---|---|
Events | No | No | None | Events that you have defined for the node are displayed on this tab. By
default, no monitoring events are defined on any node in a message flow. Use
Add, Edit, and Delete to
create, change or delete monitoring events for the node. You can enable and disable events that are shown here by selecting or clearing the Enabled check box. |
Local environment overrides
You can dynamically override set values in the local environment in the same way as setting values in other elements of a message. For more information, see Using local environment variables with Kafka nodes.
Kafka custom header properties
You can use Kafka custom header properties to add metadata to the Kafka message for use during message processing. For more information, see Setting and retrieving Kafka custom header properties.