Monitoring Amazon Web Services (AWS) with Amazon Web Services (AWS) agent

Installing on EC2

You can install the AWS agent on Windows or Linux on EC2. It is better to run the Instana AWS agent on a Current Generation General Purpose machine that runs Linux®. For example, m5.large instances are ideal.

If you install the AWS agent on Windows, even though the Windows machine is hosted in AWS, you still need to complete the configurations in the Installation outside your AWS infrastructure section.

1. Click More > Agents > Install agents > AWS in the Instana UI.

2. From the Technology list, select Instana AWS Sensor.

3. From the Run your AWS Agent on list, select Elastic Cloud Compute (EC2) (it's the default).

   Your agent key and location are pre-populated in the script to be used as User Data on EC2. Copy the script that is shown. The script looks as follows, except that all the necessary information is provided:

   #!/bin/bash
   
   curl -o setup_agent.sh https://setup.instana.io/agent
   chmod 700 ./setup_agent.sh
   sudo ./setup_agent.sh -y -a <your-agent-key> -m aws -t dynamic -e <location> -s
   

4. Spin up a dedicated EC2 virtual machine, and use the copied script as User Data. For more information on how to user User Data on EC2, see Running Commands on Your Linux Instance at Launch documentation.

5. Copy the configurations in the code block of the IAM roles topic to a configuration.json file. These configurations are used for the needed IAM role for the EC2 virtual machine that runs the Instana AWS agent and enable the AWS agent to discover and monitor your AWS resources. Then, create an IAM role with the configuration.json file. For more information, see Creating a role to delegate permissions to an IAM user.

6. For the IAM role to do the AssumedRole action, edit the Trust Relationship as follows:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

   In case your EC2 instance is in one of China's regions, edit the Trust Relationship as follows:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com.cn"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

   For additional necessary configurations, see specifics of the China region monitoring

Installation on Fargate on ECS

You can install the AWS agent on Windows or Linux on Fargate on ECS.

If you install the AWS agent on Windows, even though the Windows machine is hosted in AWS, you still need to complete the configurations in the Installation outside your AWS infrastructure section.

1. Click More > Agents > Install agents > AWS in the Instana UI.

2. From the Technology list, select Instana AWS Sensor.

3. From the Run your AWS Agent on list, select Elastic Container Service (ECS).

   The JSON of a Task Definition is pre-generated for you in the Task Definition JSON template. Download the JSON file, and use it with the Configure by using JSON functions of the Task Definition user interface.

4. Download the IAM permissions JSON file, and assign to the new Task Definition an IAM role that contains at least the permissions that are listed in the downloaded JSON file.

5. Create an ECS Service with one instance of the newly created Task Definition.

Installation outside your AWS infrastructure

You can also dedicate any agent that runs outside your AWS infrastructure. To achieve this goal, you need to specify the following environment variables within the /opt/instana/agent/bin/setenv file:

• The region that you want to monitor:

  INSTANA_AWS_REGION_CONFIG=
  

• The credentials to access AWS resources. These credentials should belong to a user, which is allowed to access the resources already described in the Amazon Web Services IAM Configuration section.

  AWS_ACCESS_KEY_ID=
  AWS_SECRET_ACCESS_KEY=
  

Proxy configuration

export HTTP_PROXY=
export HTTPS_PROXY=

com.instana.plugin.aws:
  proxy_host: 'example.com' # proxy host name or ip address
  proxy_port: 3128 # proxy port
  proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
  proxy_username: 'username' # OPTIONAL: proxy username
  proxy_password: 'password' # OPTIONAL: proxy password
  tagging: # proxy setup for AWS Tagging API, used for AWS resource monitoring filtering
    proxy_host: 'example.com' # proxy host name or ip address
    proxy_port: 3128 # proxy port
    proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
    proxy_username: 'username' # OPTIONAL: proxy username
    proxy_password: 'password' # OPTIONAL: proxy password

China region specifics

By default, the Instana agent contacts the global STS endpoint from the EC2 instance to gather instance profile credentials. Since this approach doesn't work in CN regions, the STS region endpoint needs to be displayed as an environment variable, which tells the Instana agent to use regional (CN) STS endpoints.

After the Instana agent is installed, specify the following environment variable within the <instana-agent-dir>/bin/setenv:

export AWS_STS_REGIONAL_ENDPOINTS=regional

Enable monitoring for specific services

For each service, you need to add the needed permissions on the individual service pages, linked to from the Monitored Services section. To exclude a service from monitoring, leave out the respective permission.

Alternatively, another way to exclude a service from monitoring is to add the appropriate-enabled_ flag in <agent_install_dir>/etc/instana/configuration.yml as described on the following individual service pages.

Monitoring multiple AWS accounts

The AWS Instana agent supports monitoring services from multiple AWS accounts in the same region. Currently, there are two approaches: using AWS named profiles and by using AWS Security Token Service (STS).

When you configure the Instana agent to monitor multiple AWS accounts, only one of the two available approaches should be used.

[default]
aws_access_key_id = accessKey1
aws_secret_access_key = secretAccessKey1

[profile2]
aws_access_key_id = accessKey2
aws_secret_access_key = secretAccessKey2

[profile3]
aws_access_key_id = accessKey3
aws_secret_access_key = secretAccessKey3

com.instana.plugin.aws:
  profile_names:
    - 'profile2'
    - 'profile3'

com.instana.plugin.aws:
  role_arns:
    - 'arn:aws:iam::<account_2_id>:role/<role_2_name>'
    - 'arn:aws:iam::<account_3_id>:role/<role_3_name>'

Filtering and tagging

To enable filtering, the IAM Permission tag:GetResources should be included in the IAM permissions that are granted to the AWS Agent.

After you enable the monitoring of a service, it is possible to filter, which of its instances Instana will monitor based on AWS tags by modifying the agent configuration file /opt/instana/agent/etc/instana/configuration.yaml of the AWS Agent.

Available options in the configuration file are:

com.instana.plugin.aws:
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is discovered.
  include_tags: ...
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is omitted from discovery.
  exclude_tags: ...
  # Exclude untagged AWS resources by setting the flag to `false`
  include_untagged: ...

Multiple tags can be defined, separated by a comma. Tags should be a key-value pair that is separated by :. When you define tags in both lists (include and exclude), the exclude list has higher priority. If there is no need for filtering services, the configuration should not be defined.

The Instana agent automatically monitors all AWS resources that have no tags that are assigned.{: note} To exclude untagged resources from monitoring, set the include_untagged flag to false.

Filtering can also be configured on the service level. In this case, the default configuration is overridden for the particular service. For more details about discovery filtering by specific service, see specific service documentation.

Tag poll rate

To specify how often the AWS agent polls monitored AWS service to get associated tags, use the tagged-services-poll-rate configuration property. The default is 300 seconds. The agent uses AWS service client to poll tags.

The poll rate configuration for the tagged resources is:

com.instana.plugin.aws:
  tagged-services-poll-rate: 60 #default 300

Polling interval

The polling interval indicates how often AWS agent calls the CloudWatch API. It is configurable as a cloudwatch_period in the configuration.yml file. The default value is 300 seconds. It's most common to see monitoring platforms use a value 5 - 10 minutes.

The polling interval is configurable on two levels:

• Agent level for all AWS services monitored by the agent:

  com.instana.plugin.aws:
    cloudwatch_period: 300
  

• Per AWS service:

  com.instana.plugin.aws.rds:
    cloudwatch_period: 300
  

Configuration for single service overrides agent level configuration.

CloudWatch costs

To provide insights into your AWS services, monitoring platforms like Instana must use the CloudWatch API. This API is provided by AWS with consumption-based pricing, and the purpose of this documentation is to provide transparency into how Instana uses the CloudWatch API for users to understand the impact on their AWS bill.

CloudWatch costs' are affected by:

• Number of monitored instances of AWS services
• Number of collected metrics per AWS agent
• Polling interval (configurable)

Each AWS agent calls the GetMetricData CloudWatch API request for metrics collection, and if it fails GetMetricStatistics is called as a fallback. Both endpoints incur the same costs per metrics (for clarity: the fallback usage of GetMetricStatistics does not incur extra costs). You can find details for both on the AWS CloudWatch pricing page.

The following table contains roughly estimated daily costs for different AWS agents when the polling interval is 5 minutes and Amazon charge is $0.01 per 1000 CloudWatch metrics:

Note that the CloudWatch are not used for EC2 and for polling tags, so there are no CloudWatch costs incurred there.

You can reduce CloudWatch costs by increasing the polling interval. The downside to increasing the polling interval is that you reduce granularity metrics from CloudWatch. Each request provides a single metric value, so the difference between a polling interval of 60s and 300s is five metric values or one metric value for a 300s period.

It is up to you to determine whether the default polling interval is right for your business or whether to customize it to strike the balance between cost and quality of insights.

Failed to monitor AWS services in air-gapped environments

If the Instana host agent runs on an EC2 machine that lacks internet access, the agent will fail to access public APIs of AWS services and therefore fail to monitor supported AWS services.

AWS PrivateLink provides private access to services that are hosted on AWS in a highly available and scalable manner, without a need for using public IPs and without a need for traffic to traverse the internet.

To use AWS PrivateLink, create a VPC endpoint for all AWS services that need to be monitored. The VPC endpoint provides an elastic network interface in your subnet with a private IP address to the AWS service API that can be used by the Instana host agent.

For the AWS Resource Groups Tagging API, which is used by the Instana host agent to provide filtering and tagging, no VPC endpoint can be used, and this API is reachable only through a public IP address. One of possible solutions is to set a proxy to make the AWS Resource Groups Tagging API available to the Instana host agent.