Monitoring Kafka Connect

The Kafka Connect sensor is automatically deployed and installed after you install the Instana agent.

Supported Versions

Instana supports monitoring Kafka Connect distributed and Confluent Kafka Connect distributed.

Configuration

The Instana agent automatically detects the running Kafka Connect distributed instance and therefore no configuration is required.

Agent will use configuration path provided in commandline alongside connect-distributed.sh.

Kafka Worker - Metrics collection

Configuration data

  • Version
  • Name
  • Number of brokers

Performance metrics

  • Connector Count
  • Connector Startup Failure
  • Task Startup Failure

Rebalancing

Metric Description Granularity
Completed Rebalances Total number of rebalances completed by this worker kafka.connect:type=connect-worker-rebalance-metrics,name=completed-rebalances-total. 1 second
Rebalance Average Time The average time in milliseconds spent by this worker to rebalance kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-avg-time-ms. 1 second
Rebalancing Whether this worker is currently rebalancing kafka.connect:type=connect-worker-rebalance-metrics,name=rebalancing. 1 second
Time Since Last Rebalance The time in milliseconds since this worker completed the most recent rebalance kafka.connect:type=connect-worker-rebalance-metrics,name=time-since-last-rebalance-ms. 1 second

Kafka Connector - Metrics collection

Configuration data

  • Connector Name
  • Connector Class
  • Connector Type
  • Connector Version

Connector Tasks

Metric Description Granularity
Total The number of tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-total-task-count. 1 second
Running The number of running tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-running-task-count 1 second
Failed The number of failed tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-failed-task-count. 1 second
Destroyed The number of destroyed tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-destroyed-task-count. 1 second
Paused The number of paused tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-paused-task-count. 1 second
Unassigned The number of unassigned tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-unassigned-task-count 1 second

Tasks

Metric Description Granularity
Name Task name kafka.connect:type=connector-task-metrics,connector={connectorName},task={taskName}. 1 second
Status The status of the connector task kafka.connect:type=connector-task-metrics,connector={connectorName},task={taskName},name=status. 1 second
Running Ratio The fraction time this task has spent in running state kafka.connect:type=connector-task-metrics,connector={connectorName},task={taskName},name=running-ratio. 1 second
Paused Ratio The fraction time this task has spent in pause state kafka.connect:type=connector-task-metrics,connector={connectorName},task={taskName},name=pause-ratio. 1 second

Sink Tasks

Metric Description Granularity
Partition Count The number of topic partitions assigned to this task kafka.connect:type=sink-task-metrics,connector={connectorName},task={taskName},name=partition-count. 1 second
Record Read Rate The average per-second number of records read from Kafka for this task kafka.connect:type=sink-task-metrics,connector={connectorName},task={taskName},name=sink-record-read-rate. 1 second
Record Send Rate The average per-second number of records sent from this task kafka.connect:type=sink-task-metrics,connector={connectorName},task={taskName},name=sink-record-send-rate. 1 second

Source Tasks

Metric Description Granularity
Record Poll Rate The average per-second number of records polled by this task kafka.connect:type=source-task-metrics,connector={connectorName},task={taskName},name=source-record-poll-rate. 1 second
Record Write Rate The average per-second number of records written to Kafka by this task kafka.connect:type=source-task-metrics,connector={connectorName},task={taskName},name=source-record-write-rate. 1 second

Kafka Connect Cluster - Metrics collection

Configuration data

  • Version

Workers

Metric Description Granularity
Connector Startup Failure Total number of rebalances completed by this worker kafka.connect:type=connect-worker-rebalance-metrics,name=connector-startup-failure-total. 1 second
Task Startup Failure Total number of rebalances completed by this worker kafka.connect:type=connect-worker-rebalance-metrics,name=task-startup-failure-total. 1 second
Completed Rebalances Total number of rebalances completed by this worker kafka.connect:type=connect-worker-rebalance-metrics,name=completed-rebalances-total. 1 second
Rebalance Average Time The average time in milliseconds spent by this worker to rebalance kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-avg-time-ms. 1 second
Rebalancing Whether this worker is currently rebalancing kafka.connect:type=connect-worker-rebalance-metrics,name=rebalancing. 1 second
Time Since Last Rebalance The time in milliseconds since this worker completed the most recent rebalance kafka.connect:type=connect-worker-rebalance-metrics,name=time-since-last-rebalance-ms. 1 second

Connectors

Metric Description Granularity
Name Connector name kafka.connect:type=connect-worker-metrics,connector={name}. 1 second
Total Tasks The number of tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-total-task-count. 1 second
Running Tasks The number of running tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-running-task-count. 1 second
Failed Tasks The number of failed tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-failed-task-count. 1 second
Destroyed Tasks The number of destroyed tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-destroyed-task-count. 1 second
Paused Tasks The number of paused tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-paused-task-count. 1 second
Unassigned Tasks The number of unassigned tasks of the connector kafka.connect:type=connect-worker-metrics,connector={name},name=connector-unassigned-task-count. 1 second