Monitoring Kafka

All Kafka metrics that Instana collects are available for every version of Apache Kafka, Cloudera Kafka, and Confluent Kafka, except the Consumer group lag and the Consumer/Producer Byte Rate/Throttling metrics. IBM® Event Streams, which is built on open source Apache Kafka, is supported from IBM Event Streams 11.0.4 (IBM Event Streams Operator 3.0.5) and later versions.

The following table shows the latest supported version and support policy:

For more information about the support policy, see Support strategy for sensors.

Consumer group lag metrics are available for the following versions:

• Apache Kafka (with Zookeeper) versions from 0.11.x.x to 3.9.0
• Apache Kafka (with KRaft) versions from 3.9.0
• Cloudera Kafka versions from 3.x.x to 4.1.x
• Confluent Kafka versions from 3.3.x
• IBM Event Streams version from 11.0.4 (IBM Event Streams Operator 3.0.5) and later versions

Consumer/Producer Byte Rate/Throttling metrics are available for Java Kafka clients only and for the following versions:

• Apache Kafka versions from 1.1.x
• Cloudera Kafka versions from 4.0.x to 4.1.x
• Confluent Kafka versions from 4.1.x
• IBM Event Streams version from 11.0.4 (IBM Event Streams Operator v3.0.5) and later versions

Instana supports client-side tracing for the following languages and runtimes:

• Java
• Go
• Node.js
• .Net Core

You can configure how often Instana polls Kafka to collect data and metrics by using the poll_rate parameter in the agent configuration.yaml file as shown in the following example:

If your Kafka broker instance requires SSL client connections, you need to configure the Instana agent via <agent_install_dir>/etc/instana/configuration.yaml to enable collecting Consumer lag metrics as shown in the following example:

com.instana.plugin.kafka:
  ...
  sslTrustStore: '/path/to/truststore.jks'
  sslTrustStorePassword: 'kafkaTsPassword'
  sslKeyStore: '/path/to/sslKeyStoreFile.jks'
  sslKeyStorePassword: 'kafkaKsPassword'
 

If your Kafka broker instance requires SASL or PLAIN authentication, configure the Instana agent through <agent_install_dir>/etc/instana/configuration.yaml to enable collection of Consumer lag metrics as shown in the following example:

com.instana.plugin.kafka:
  ...
  saslUsername: 'kafkaUser'
  saslPassword: 'kafkaPassword'
 

Make sure that the Keys are in the Java Keystore (JKS) format. Use the keytool to create the keys.

If your JMX authentication is enabled for your Kafka, you need to configure the Instana agent by using <agent_install_dir>/etc/instana/configuration.yaml to authenticate your JMX as shown in the following example:

com.instana.plugin.kafka:
  jmxUsername: ''
  jmxPassword: ''
  jmxPort: '' # default jmx port is 1099
 

You can configure the format for Kafka trace correlation headers that are used by Instana tracers. For more information, see Configuring Kafka trace correlation headers.

You need the following details to configure Kafka node:

• Version
• Zookeeper Connects
• Process ID
• Node ID
• Topics/Partitions

The following table contains the performance metrics details:

You need the following details to configure Kafka cluster:

• Cluster Name
• Zookeeper
• Nodes (Name, Version)
• Topics/Partitions

The following table contains the performance metrics details:

For each sensor, a knowledge base of health signatures are evaluated continuously against the incoming metrics and are used to raise issues or incidents depending on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any given entity.

For more information about built-events for Kafka node and cluster, see Built-in events reference.

Monitoring issue type: kafka_ssl_not_configured

To resolve the SSL configuration issue and configure Kafka SSL truststore location and password, see SSL/TLS Support.

Monitoring issue type: kafka_ssl_client_not_configured

To resolve the SSL client authentication related issue and configure Kafka SSL client authentication (keystore location and password), see SSL/TLS Support.

Monitoring issue type: kafka_invalid_jmx_credentials

To resolve the JMX authentication related issue and configure JMX authentication credentials, see JMX Authentication support.