Monitoring Amazon Web Services (AWS) with Amazon Web Services (AWS) agent

To monitor services that are managed by AWS, Instana needs to collect data from AWS APIs, such as CloudWatch, S3, and X-Ray, for which it is not possible to install a host agent or any other type of agent.

However, the Instana host agent can be set up in specific ways to monitor services that are managed by AWS. The host agent that is set up in specific ways to monitor AWS services is called AWS agent. You are recommended to install the AWS agent on a dedicated host machine. Also, make sure that at least one AWS agent is available in each AWS region.

After you install the AWS agent, you can get data in and out of the supported AWS services, but you do not have access to the infrastructures that powers the services.

Notes:

  • To monitor an Amazon Elastic Computing (EC2) virtual machine, a Kubernetes cluster that runs on AWS, either installed and managed by you or using Amazon Elastic Kubernetes Service, or an Amazon Elastic Container cluster, you need to use the Instana host agent, instead of AWS agent. For more information about the host agent installation steps, see the topics in the Platforms section.

  • To monitor AWS in a Kubernetes or Red Hat OpenShift cluster, do not install Instana AWS agent on each node of the cluster. Install AWS agent on a dedicated host machine.

Platforms

To monitor the following platforms, install the Instana host agent:

Monitored services

To monitor the following services that are managed by AWS, install the Instana AWS agent as described in the Installing AWS agent section:

Due to the nature of the AWS CloudWatch API, a delay in metrics retrieval is possible. The displayed values for the metrics are also delayed ensuring consistency. This delay depends on the actual AWS service and availability of data then but usually is around 10 minutes.

Installing AWS agent

You can install the AWS agent on EC2 or on Fargate on ECS.

Notes:

  • Depending on the number of monitored entities in your cloud environment, you might need to increase the maximum amount of available memory to your host agent. The agent memory can be increased by setting the environment variable AGENT_MAX_MEM to a value that is greater than the default value 512 MB. For example, to set the agent memory to 1 GB, you can set AGENT_MAX_MEM=1024mb.

  • Install only one AWS agent per combination of AWS account and AWS region. Installing multiple AWS agents for the same combination of AWS account and AWS region can incur extra costs from AWS, without added benefit in terms of quality of monitoring by using Instana.

Installing on EC2

You can install the AWS agent on Windows or Linux on EC2. It is better to run the Instana AWS agent on a Current Generation General Purpose machine that runs Linux®. For example, m5.large instances are ideal.

If you install the AWS agent on Windows, even though the Windows machine is hosted in AWS, you still need to complete the configurations in the Installation outside your AWS infrastructure section.

  1. Click More > Agents > Install agents > AWS in the Instana UI.

  2. From the Technology list, select Instana AWS Sensor.

  3. From the Run your AWS Agent on list, select Elastic Cloud Compute (EC2) (it's the default).

    Your agent key and location are pre-populated in the script to be used as User Data on EC2. Copy the script that is shown. The script looks as follows, except that all the necessary information is provided:

    #!/bin/bash
    
    curl -o setup_agent.sh https://setup.instana.io/agent
    chmod 700 ./setup_agent.sh
    sudo ./setup_agent.sh -y -a <your-agent-key> -m aws -t dynamic -e <location> -s
    
  4. Spin up a dedicated EC2 virtual machine, and use the copied script as User Data. For more information on how to user User Data on EC2, see Running Commands on Your Linux Instance at Launch documentation.

  5. Copy the configurations in the code block of the IAM roles topic to a configuration.json file. These configurations are used for the needed IAM role for the EC2 virtual machine that runs the Instana AWS agent and enable the AWS agent to discover and monitor your AWS resources. Then, create an IAM role with the configuration.json file. For more information, see Creating a role to delegate permissions to an IAM user.

  6. For the IAM role to do the AssumedRole action, edit the Trust Relationship as follows:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    

    In case your EC2 instance is in one of China's regions, edit the Trust Relationship as follows:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com.cn"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    

    For additional necessary configurations, see specifics of the China region monitoring

Installation on Fargate on ECS

You can install the AWS agent on Windows or Linux on Fargate on ECS.

If you install the AWS agent on Windows, even though the Windows machine is hosted in AWS, you still need to complete the configurations in the Installation outside your AWS infrastructure section.

  1. Click More > Agents > Install agents > AWS in the Instana UI.

  2. From the Technology list, select Instana AWS Sensor.

  3. From the Run your AWS Agent on list, select Elastic Container Service (ECS).

    The JSON of a Task Definition is pre-generated for you in the Task Definition JSON template. Download the JSON file, and use it with the Configure by using JSON functions of the Task Definition user interface.

  4. Download the IAM permissions JSON file, and assign to the new Task Definition an IAM role that contains at least the permissions that are listed in the downloaded JSON file.

  5. Create an ECS Service with one instance of the newly created Task Definition.

Configuration

Installation outside your AWS infrastructure

You can also dedicate any agent that runs outside your AWS infrastructure. To achieve this goal, you need to specify the following environment variables within the /opt/instana/agent/bin/setenv file:

  • The region that you want to monitor:

    INSTANA_AWS_REGION_CONFIG=
    
  • The credentials to access AWS resources. These credentials should belong to a user, which is allowed to access the resources already described in the Amazon Web Services IAM Configuration section.

    AWS_ACCESS_KEY_ID=
    AWS_SECRET_ACCESS_KEY=
    

Proxy configuration

Through environment variables

To configure the AWS Sensor to use proxy configuration, specify the following environment variables within the <instana-agent-dir>/bin/setenv:

export HTTP_PROXY=
export HTTPS_PROXY=

Both environment variables must be defined. After it is defined, Instana agent needs to be rebooted for changes to take place.

You can find more information about the HTTP proxy on the link.

Through agent configuration.yaml

To configure AWS agent to use proxy configuration, add the following agent configuration settings:

com.instana.plugin.aws:
  proxy_host: 'example.com' # proxy host name or ip address
  proxy_port: 3128 # proxy port
  proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
  proxy_username: 'username' # OPTIONAL: proxy username
  proxy_password: 'password' # OPTIONAL: proxy password
  tagging: # proxy setup for AWS Tagging API, used for AWS resource monitoring filtering
    proxy_host: 'example.com' # proxy host name or ip address
    proxy_port: 3128 # proxy port
    proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
    proxy_username: 'username' # OPTIONAL: proxy username
    proxy_password: 'password' # OPTIONAL: proxy password

Proxy settings can also be configured at the level of individual AWS Sensor. In this case, the previously mentioned global configuration is overridden by the specific AWS Sensor configuration. For more information, see the specific AWS Sensor documentation.

China region specifics

By default, the Instana agent contacts the global STS endpoint from the EC2 instance to gather instance profile credentials. Since this approach doesn't work in CN regions, the STS region endpoint needs to be displayed as an environment variable, which tells the Instana agent to use regional (CN) STS endpoints.

After the Instana agent is installed, specify the following environment variable within the <instana-agent-dir>/bin/setenv:

export AWS_STS_REGIONAL_ENDPOINTS=regional

Enable monitoring for specific services

For each service, you need to add the needed permissions on the individual service pages, linked to from the Monitored Services section. To exclude a service from monitoring, leave out the respective permission.

Alternatively, another way to exclude a service from monitoring is to add the appropriate-enabled_ flag in <agent_install_dir>/etc/instana/configuration.yml as described on the following individual service pages.

Monitoring multiple AWS accounts

The AWS Instana agent supports monitoring services from multiple AWS accounts in the same region. Currently, there are two approaches: using AWS named profiles and by using AWS Security Token Service (STS).

When you configure the Instana agent to monitor multiple AWS accounts, only one of the two available approaches should be used.

AWS named profiles approach

To monitor multiple accounts, you need to define named profiles in the same way you would with the AWS CLI, or creating the file manually. The .aws/credentials file must be created within the home directory of the user who is running the Instana agent, which is usually root. The AWS CLI uses credentials files ~/.aws/credentials as follows:

[default]
aws_access_key_id = accessKey1
aws_secret_access_key = secretAccessKey1

[profile2]
aws_access_key_id = accessKey2
aws_secret_access_key = secretAccessKey2

[profile3]
aws_access_key_id = accessKey3
aws_secret_access_key = secretAccessKey3

Every profile represents the mapping between AWS named profile and AWS access credentials. Upon installation of an AWS agent, a default AWS profile is created. There is no need to add the default profile to ~/.aws/credentials file.

The additional profiles to be used by the AWS agent must be listed as follows in the agent configuration file /opt/instana/agent/etc/instana/configuration.yaml:

com.instana.plugin.aws:
  profile_names:
    - 'profile2'
    - 'profile3'

Notes:

  • The monitoring of multiple AWS accounts can also be specified for particular AWS services within their specific configuration. The list of profiles that are specified for a specific service override the general configurations.
  • Be sure to follow recommendations when it comes to the creation of the credentials file.

To avoid any security threats, you should either set more strict rules for EC2 instance or create a separate Instana AWS user and that way avoid adding root user security credentials to the ~/.aws/credentials file.

AWS STS approach

This approach uses AWS STS service API to obtain access credentials for all additional AWS accounts that should be monitored by the Instana agent. After the agent is installed and configured to monitor default account, following the steps that are described in the Installation section, you should provide IAM Role ARNs from all additional AWS accounts, in the following way:

com.instana.plugin.aws:
  role_arns:
    - 'arn:aws:iam::<account_2_id>:role/<role_2_name>'
    - 'arn:aws:iam::<account_3_id>:role/<role_3_name>'

The monitoring of multiple AWS accounts can also be specified for particular AWS services within their specific configuration. The list of IAM roles that are specified for a specific service override the general configurations.

Each role, matching the specified ARN, should also comprise the needed IAM permissions necessary for Instana AWS monitoring. Thus, each role should allow default account to do sts:AssumeRole by specifying the Trust relationship policy in the following way:

  1. If sts:AssumeRole is done from the AWS user context - when the agent is installed outside your AWS infrastructure

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::<default_account_id>:user/<default_account_username>"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    
  2. If sts:AssumeRole is done from the IAM Role context - when the agent is installed on EC2:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::<default_account_id>:role/<default_account_IAM_role_name>"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    

    Similarly, default account should be also allowed to do sts:AssumeRole to all additional accounts. This can be done by specifying extra IAM policies within the IAM Role that is used for Instana monitoring, in the following way:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            #Instana monitoring policy specifications
          },
          {
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::<account_1_id>:role/<role_1_name>",
            "Effect": "Allow"
          },
          {
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::<account_2_id>:role/<role_2_name>",
            "Effect": "Allow"
          }
        ]
      }
    

Filtering and tagging

To enable filtering, the IAM Permission tag:GetResources should be included in the IAM permissions that are granted to the AWS Agent.

After you enable the monitoring of a service, it is possible to filter, which of its instances Instana will monitor based on AWS tags by modifying the agent configuration file /opt/instana/agent/etc/instana/configuration.yaml of the AWS Agent.

Available options in the configuration file are:

com.instana.plugin.aws:
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is discovered.
  include_tags: ...
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is omitted from discovery.
  exclude_tags: ...
  # Exclude untagged AWS resources by setting the flag to `false`
  include_untagged: ...

Multiple tags can be defined, separated by a comma. Tags should be a key-value pair that is separated by :. When you define tags in both lists (include and exclude), the exclude list has higher priority. If there is no need for filtering services, the configuration should not be defined.

The Instana agent automatically monitors all AWS resources that have no tags that are assigned.{: note} To exclude untagged resources from monitoring, set the include_untagged flag to false.

Filtering can also be configured on the service level. In this case, the default configuration is overridden for the particular service. For more details about discovery filtering by specific service, see specific service documentation.

Tag poll rate

To specify how often the AWS agent polls monitored AWS service to get associated tags, use the tagged-services-poll-rate configuration property. The default is 300 seconds. The agent uses AWS service client to poll tags.

The poll rate configuration for the tagged resources is:

com.instana.plugin.aws:
  tagged-services-poll-rate: 60 #default 300

Polling interval

The polling interval indicates how often AWS agent calls the CloudWatch API. It is configurable as a cloudwatch_period in the configuration.yml file. The default value is 300 seconds. It's most common to see monitoring platforms use a value 5 - 10 minutes.

The polling interval is configurable on two levels:

  • Agent level for all AWS services monitored by the agent:

    com.instana.plugin.aws:
      cloudwatch_period: 300
    
  • Per AWS service:

    com.instana.plugin.aws.rds:
      cloudwatch_period: 300
    

Configuration for single service overrides agent level configuration.

CloudWatch costs

To provide insights into your AWS services, monitoring platforms like Instana must use the CloudWatch API. This API is provided by AWS with consumption-based pricing, and the purpose of this documentation is to provide transparency into how Instana uses the CloudWatch API for users to understand the impact on their AWS bill.

CloudWatch costs' are affected by:

  • Number of monitored instances of AWS services
  • Number of collected metrics per AWS agent
  • Polling interval (configurable)

Each AWS agent calls the GetMetricData CloudWatch API request for metrics collection, and if it fails GetMetricStatistics is called as a fallback. Both endpoints incur the same costs per metrics (for clarity: the fallback usage of GetMetricStatistics does not incur extra costs). You can find details for both on the AWS CloudWatch pricing page.

The following table contains roughly estimated daily costs for different AWS agents when the polling interval is 5 minutes and Amazon charge is $0.01 per 1000 CloudWatch metrics:

AWS sensor Number of metrics Daily cost per instance Note
API Gateway 5 - 11 $0.0144 - $0.0317 The number of metrics depends on the API protocol.
AppSync 18 $0.0518
Auto Scaling 13 $0.0374
Beanstalk 31 $0.0893
CloudFront 13 $0.0374 The number of metrics depends on the number of associated functions. For example, the table shows one associated function for the monitored distribution.
DynamoDB 71 $0.2045
EBS 9 $0.0230
ElastiCache 25 - 39 $0.0720 - $0.1123 The number of metrics depends on the engine used.
OpenSearch 12 $0.0346
ELB 5 - 15 $0.0144 - $0.0432 The number of metrics depends on the load balancer type and number of availability zones.
EMR 15 $0.0432
IoT Core 23 $0.0662
Kinesis 16 $0.0461
Lambda 21 $0.0605
MQ 22 $0.0634
MSK 37 $0.0844 The number of metrics depends on the number of brokers within a cluster. For example, the table shows that one broker is monitored.
RDS 18 $0.0518
Redshift 17 $0.0490 The number of metrics depends on the number of nodes within a cluster. For example, the table shows that one node is monitored.
S3 13 $0.0374
SNS 7 $0.0202
SQS 9 $0.0230
Timestream 9 $0.0230

Note that the CloudWatch are not used for EC2 and for polling tags, so there are no CloudWatch costs incurred there.

You can reduce CloudWatch costs by increasing the polling interval. The downside to increasing the polling interval is that you reduce granularity metrics from CloudWatch. Each request provides a single metric value, so the difference between a polling interval of 60s and 300s is five metric values or one metric value for a 300s period.

It is up to you to determine whether the default polling interval is right for your business or whether to customize it to strike the balance between cost and quality of insights.

Troubleshooting

Failed to monitor AWS services in air-gapped environments

If the Instana host agent runs on an EC2 machine that lacks internet access, the agent will fail to access public APIs of AWS services and therefore fail to monitor supported AWS services.

AWS PrivateLink provides private access to services that are hosted on AWS in a highly available and scalable manner, without a need for using public IPs and without a need for traffic to traverse the internet.

To use AWS PrivateLink, create a VPC endpoint for all AWS services that need to be monitored. The VPC endpoint provides an elastic network interface in your subnet with a private IP address to the AWS service API that can be used by the Instana host agent.

For the AWS Resource Groups Tagging API, which is used by the Instana host agent to provide filtering and tagging, no VPC endpoint can be used, and this API is reachable only through a public IP address. One of possible solutions is to set a proxy to make the AWS Resource Groups Tagging API available to the Instana host agent.