Monitoring Amazon MSK
The Amazon MSK sensor is automatically deployed and installed after you install the Instana agent.
Learn about the other supported AWS services with our AWS documentation.
- Sensor (Data Collection)
- MSK Broker - Metrics collection
- MSK Cluster - Metrics collection
- Required Permissions
- Known limitations
Sensor (Data Collection)
MSK Broker - Metrics collection
Tracked Configuration
| MSK Instance Details | Description |
|---|---|
| ID | ID of specific MSK instance. |
| ARN | The unqualified Amazon Resource Name (ARN) assigned to the instance. |
| Type | MSK instance type. |
| Instance Type | Instance tier type. |
| Added to cluster | Time when broker was added to cluster. |
Performance metrics
| Performance metric | Description |
|---|---|
| Partitions | Number of partitions. |
| Under-replicated Partition Count | Number of under-replicated partitions. |
Metrics
| Broker Metrics | Description | Monitoring Level |
|---|---|---|
| Leader Count | The total number of leaders of partitions per broker, not including replicas. | DEFAULT |
| Bytes In | The number of bytes per second received from clients. This metric is available per broker and also per topic. | DEFAULT |
| Bytes Out | The number of bytes per second sent to clients. This metric is available per broker and also per topic. | DEFAULT |
| Messages In | The number of incoming messages per second for the broker. | DEFAULT |
| Request Bytes | The mean number of request bytes for the broker. | DEFAULT |
| CPU Idle | The percentage of CPU idle time. | DEFAULT |
| CPU User | The percentage of CPU in user space. | DEFAULT |
| CPU System | The percentage of CPU in kernel space. | DEFAULT |
| Memory Free | The size in bytes of memory that is free and available for the broker. | DEFAULT |
| Memory Used | The size in bytes of memory that is in use for the broker. | DEFAULT |
| Memory Cached | The size in bytes of cached memory for the broker. | DEFAULT |
| Memory Buffered | The size in bytes of buffered memory for the broker. | DEFAULT |
| Swap Free | The size in bytes of swap memory that is available for the broker. | DEFAULT |
| Swap Used | The size in bytes of swap memory that is in use for the broker. | DEFAULT |
| Network Rx Received | The number of packets received by the broker. | DEFAULT |
| Network Rx Dropped | The number of dropped receive packages. | DEFAULT |
| Network Rx Errors | The number of network receive errors for the broker. | DEFAULT |
| Network Tx Transmitted | The number of packets transmitted by the broker. | DEFAULT |
| Network Rx Dropped | The number of dropped transmit packages. | DEFAULT |
| Network Rx Errors | The number of network transmit errors for the broker. | DEFAULT |
| Produce Time | The mean produce time in milliseconds. | PER_BROKER |
| Produce Throttle Time | The average produce throttle time in milliseconds. | PER_BROKER |
| Fetch Throttle Time | The average fetch throttle time in milliseconds. | PER_BROKER |
| Request Throttle Time | The average request throttle time in milliseconds. | PER_BROKER |
| Produce Throttle Byte Rate | The number of throttled bytes per second. | PER_BROKER |
| Fetch Throttle Byte Rate | The number of throttled bytes per second. | PER_BROKER |
| Consumer Fetch Time | The mean total time in milliseconds that consumers spend on fetching data from the broker. | PER_BROKER |
| Follower Fetch Time | The mean total time in milliseconds that followers spend on fetching data from the broker. | PER_BROKER |
| Network Idle Time | The average percentage of the time the network processors are idle. | PER_BROKER |
| Request Idle Time | The average percentage of the time the request handler threads are idle. | PER_BROKER |
MSK Cluster - Metrics collection
Tracked Configuration
| MSK Cluster Details | Description |
|---|---|
| Name | Name of MSK cluster. |
| ARN | The unqualified Amazon Resource Name (ARN) assigned to the instance. |
| Enhanced Monitoring | Monitoring level. |
| Kafka Version | Version of Kafka instances. |
Performance metrics
| Performance metric | Description |
|---|---|
| Active Controllers | Number of active controllers. |
| Topics | Number of topics. |
Metrics
| Broker Metrics | Description | Monitoring Level |
|---|---|---|
| Partitions Count | The total number of partitions. | DEFAULT |
| Offline Partitions | The total number of offline partitions. | DEFAULT |
| Disk Used | Disk space usage. | DEFAULT |
Required Permissions
cloudwatch:GetMetricDatacloudwatch:GetMetricStatisticskafka:ListClusterskafka:ListNodeskafka:ListTagsForResourcekafka:DescribeCluster
Known limitations
The Amazon MSK sensor supports monitoring only for the provisioned MSK cluster type. The serverless cluster type is not supported.
Configuration
Metrics for Amazon MSK are pulled every 5 minutes, this can be changed via agent configuration (<agent_install_dir>/etc/instana/configuration.yaml):
com.instana.plugin.aws.msk:
cloudwatch_period: 300
To disable monitoring of MSK instances use the following configuration:
com.instana.plugin.aws.msk:
enabled: false
Proxy configuration
To configure the specific AWS Sensor to use proxy configuration, add the following agent configuration settings:
com.instana.plugin.aws.msk:
proxy_host: 'example.com' # proxy host name or ip address
proxy_port: 3128 # proxy port
proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
proxy_username: 'username' # OPTIONAL: proxy username
proxy_password: 'password' # OPTIONAL: proxy password
Monitoring multiple AWS accounts
Refer to the Monitoring multiple AWS accounts documentation to set up monitoring of multiple AWS accounts with one AWS agent in the same region.
AWS named profiles approach
To override which profiles should be used to monitor MSK, use the following configuration:
com.instana.plugin.aws.msk:
profile_names:
- 'profile2'
- 'profile3'
Defining profiles on service level will override the global AWS profiles configuration.
AWS STS approach
To override which IAM Roles should be used to monitor MSK, use the following configuration:
com.instana.plugin.aws.msk:
role_arns:
- 'arn:aws:iam::<account_1_id>:role/<role_1_name>'
- 'arn:aws:iam::<account_2_id>:role/<role_2_name>'
Defining IAM roles on service level will override the global AWS IAM roles configuration.
Filtering
Multiple tags can be defined, separated by a comma. Tags should be provided as a key-value pair separated by :. In order to make configuration easier, it is possible to define which tags you want to include in discovery or exclude from discovery. In case of defining tag in both lists (include and exclude), exclude list has higher priority. If there is no need for services filtering, the configuration should not be defined. It’s not mandatory to define all values in order to enable filtering.
Users are able to specify how often sensors will poll the AWS tagged resources using the tagged-services-poll-rate configuration property (default 300 seconds).
Tags are only available in conjunction with the AWS Agent.
To define how often sensors will poll the tagged resources use following configuration:
com.instana.plugin.aws:
tagged-services-poll-rate: 60 #default 300
To include services by tags into discovery use following configuration:
com.instana.plugin.aws.msk:
include_tags: # Comma separated list of tags in key:value format (e.g. env:prod,env:staging)
To exclude services by tags from discovery use following configuration:
com.instana.plugin.aws.msk:
exclude_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)
AWS services without tags will be monitored by default but can be excluded by setting the include_untagged field to false:
com.instana.plugin.aws.beanstalk:
include_untagged: false # True value by default
Instana Agent Tags
Note that tags are only available in conjunction with the AWS Agent. More details on using tags are described here.