KafkaRead node
Use the KafkaRead node in a message flow to read a single message from a Kafka topic. The message to be read is identified by the topic name, partition number, and message offset properties, which you can set in the node or as Local Environment overrides.
This topic contains the following sections:
Purpose
You can use a KafkaRead node in a message flow to read a specific message from a Kafka topic, which can be useful for error processing in a Kafka flow. For example, if the processing of a message fails in a Kafka consumer message flow, you can use the KafkaRead node to reprocess that individual message. You can also use a KafkaRead node in conjunction with a Timer node, to control the processing of a range of messages in a partition on a topic. For more information, see Reading an individual message from a Kafka topic.
For information about the supported versions of Kafka, see IBM App Connect Enterprise system requirements. For more information about Kafka version compatibility, see the Apache Kafka documentation.
The KafkaRead node handles messages in the following message domains:
- DFDL
- XMLNSC
- JSON
- BLOB
- MIME
- XMLNS
- MRM
The KafkaRead node is contained in the Kafka drawer of the palette, and is represented in the IBM App Connect Enterprise Toolkit by the following icon:
Using the KafkaRead node in a message flow
Use the KafkaRead node in a message flow to read an individual message that was published to a topic on a Kafka server. For information about how to use the node, see Reading an individual message from a Kafka topic.
Terminals and properties
The KafkaRead node terminals are described in the following table.
Terminal | Description |
---|---|
Failure | The output terminal to which the message is routed if a failure is detected during processing in the node. |
Out | The output terminal to which the message is routed if it represents successful completion of the Kafka request, and if further processing is required within this message flow. |
No match | The output terminal to which the input message is routed if no message is available on the topic. The output message that is propagated to the No match terminal is constructed from the input message. |
The following tables describe the node properties. The column headed M indicates whether the property is mandatory (marked with an asterisk on the panel if you must enter a value when no default is defined); the column headed C indicates whether the property is configurable (you can change the value when you add the message flow to the BAR file to deploy it).
The KafkaRead node Description properties are described in the following table.
Property | M | C | Default | Description |
---|---|---|---|---|
Node name | No | No | The node type, KafkaRead | The name of the node. |
Short description | No | No | A brief description of the node. | |
Long description | No | No | Text that describes the purpose of the node in the message flow. |
The KafkaRead node Basic properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Topic name | Yes | Yes | The name of the Kafka topic containing the message that you want to read. Only one topic name can be specified. The name can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). | topicName | |
Bootstrap servers | Yes | Yes | A list of host/port pairs, separated by spaces, to use for establishing the initial connection to the Kafka cluster. | bootstrapServers | |
Partition number | Yes | Yes | 0 | The number of the Kafka partition for the topic that you want to use (between 0-255). The default value is 0. | partitionNumber |
Message offset | Yes | Yes | 0 | The offset position of the message to be read. The default value is 0. | offset |
Action when message unavailable | Yes | Yes | no match | The action that should be taken if the message at the specified offset is
unavailable. Possible values are:
|
notFoundAction |
Client ID | No | Yes | The client name to be used when connecting to Kafka. This value can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). | clientId | |
Add server name suffix to client ID | No | Yes | Yes | If selected, this property suffixes the provided client ID with either
-integrationServerName (for an independent integration server)
or -integrationNodeName-integrationServerName (for an
integration server that is managed by an integration node). This option is selected by
default. |
useClientIdSuffix |
Timeout (s) | Yes | Yes | 5 | The timeout (in seconds) for the KafkaRead node to get the message. If the message has not been found within that time, the action taken is dependent on the value set by the notFoundAction property. The default is 5 seconds. | connectionTimeout |
The KafkaRead node Advanced properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Properties file | No | Yes | null | Full path name of a kafka.properties file to set Kafka client properties. | propertiesFilePath |
The KafkaRead node Security properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Security identifier | No | Yes | null | Identifier that is used to look up the user ID and password from the App Connect Enterprise vault, which was defined by using the mqsicredentials command. If this property is left blank, the
default |
securityIdentity |
Security protocol | Yes | Yes | PLAINTEXT | The protocol to be used when communicating with the Kafka server. Valid values
are:
If you are using IBM Event Streams, this property must be set to SASL_SSL. If either SASL_PLAINTEXT or SASL_SSL is selected, you must configure the user ID and password that will be used to authenticate with the Kafka server. You can configure these credentials by using either the mqsisetdbparms command or the mqsicredentials command. For more information, see Configuring security credentials for connecting to Kafka. |
securityProtocol |
SSL protocol | Yes | Yes | TLSv1_2 | The SSL protocol that is used when the selected value for the Security protocol property is either SSL or SASL_SSL.
You can select one of the following values from the editable list, or you can specify an alternative
value:
|
sslProtocol |
The KafkaRead node Policy properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Policy | No | Yes | null | Name of a Kafka policy to be used to override node properties; for example,
{PolicyProject}:kafkaPolicy
|
policyUrl |
Property | M | C | Default | Description |
---|---|---|---|---|
Output data location | No | No | $OutputRoot |
The location on the output message tree to which the KafkaRead node sends output. The input root is first copied to the output root, and then the result data is copied to the location on the output tree specified by Output data location. The default value, $OutputRoot, replaces the copied message tree with the result data, and propagates none of the input message. For example, a value of $OutputRoot/XMLNS/MyData inserts the result message data into the output message tree, by merging or overwriting the message tree. You can specify Output data location as an ESQL or an XPATH expression. See Combining a result message with an input message when fetching data from external systems. |
Result data location | No | No | $ResultRoot | The location in the message received from the Kafka queue that is copied to
the Output data location field in the output
message. For example, the default, $ResultRoot, inserts the result message into the output tree at the location specified by the Output data location. You can specify Result data location as an ESQL or an XPATH expression. See Combining a result message with an input message when fetching data from external systems. |
Copy local environment | No | No | Selected |
This property specifies whether the local environment is copied to the output message.
|
The KafkaRead node Input Message Parsing properties are described in the following table.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Message domain | No | No | BLOB | The parser domain that is used to parse the input message. If the field is blank, the default is BLOB. | |
Message model | No | No | Cleared | The name or location of the message model schema file in which the message is
defined. When you click Browse, you see a list of available message model schema files for the selected Message domain. |
|
Message | No | No | Cleared | The name or location of the message root within your message model schema file. This list is populated with all available messages that are defined in the Message model that you selected. | |
Physical format | No | No | Cleared | The name of the physical format of the message. If you are using the MRM or IDOC parser, select the physical format of the incoming message from the list. This list includes all the physical formats that you have defined for the selected message model. If you set the Message domain property to DataObject, you can set this property to XML or SAP ALE IDoc. Set this property to SAP ALE IDoc when you have to parse a bit stream from an external source and generate a message tree. |
The KafkaRead node Parser Options properties are described in the following table.
Property | M | C | Default | Description |
---|---|---|---|---|
Parse timing | No | No | On Demand | This property controls when an input message is parsed. Valid values are
On Demand, Immediate, and Complete. For a full description of this property, see Parsing on demand. |
Build tree using XML schema data types | No | No | Cleared | This property controls whether the syntax elements in the message tree have data types taken from the XML schema. |
Use XMLNSC compact parser for XMLNS domain | No | No | Cleared | This property specifies whether the XMLNSC Compact Parser is used for messages
in the XMLNS domain. If you set this property, the message data is displayed under XMLNSC in nodes
that are connected to the output terminal when the input MQRFH2 header is
XMLNS . |
Retain mixed content | No | No | Cleared | This property controls whether the XMLNSC parser creates elements in the message tree for mixed text in an input message. If you select the check box, elements are created for mixed text. If you clear the check box, mixed text is ignored and no elements are created. |
Retain comments | No | No | Cleared | This property specifies whether the XMLNSC parser creates elements in the message tree for comments in an input message. If you select the check box, elements are created for comments. If you clear the check box, comments are ignored and no elements are created. |
Retain processing instructions | No | No | Cleared | This property controls whether the XMLNSC parser creates elements in the message tree for processing instructions in an input message. If you select the check box, elements are created for processing instructions. If you clear the check box, processing instructions are ignored and no elements are created. |
Opaque elements | No | No | Blank | This property is used to specify a list of elements in the input message that are to be opaquely parsed by the XMLNSC parser. Opaque parsing is performed only if validation is not enabled (that is, if Validate is None); entries that are specified in Opaque Elements are ignored if validation is enabled. |
The KafkaRead node Validation properties are described in the following table.
For a full description of these properties see Validation properties.
Property | M | C | Default | Description | mqsiapplybaroverride command property |
---|---|---|---|---|---|
Validate | No | Yes | None | This property controls whether validation takes place. Valid values are None, Content and Value, and Content. | validateMaster |
Validation failure action | No | Yes | Exception | This property controls what happens if validation fails. You can set this property only if you set Validate to Content or Content and Value. Valid values are User Trace, Local Error Log, Exception, and Exception List. | validateFailureAction |
Property | M | C | Default | Description |
---|---|---|---|---|
Events | No | No | None | Events that you have defined for the node are displayed on this tab. By
default, no monitoring events are defined on any node in a message flow. Use
Add, Edit, and Delete to
create, change or delete monitoring events for the node. You can enable and disable events that are shown here by selecting or clearing the Enabled check box. |
Local environment overrides
You can view values for Kafka topics that have been received by the KafkaRead node. For more information, see Using local environment variables with Kafka nodes.
Kafka custom header properties
You can use Kafka custom header properties to add metadata to the Kafka message for use during message processing. For more information, see Setting and retrieving Kafka custom header properties.