Monitoring Amazon Web Services (AWS) with Amazon Web Services (AWS) agent

Installation on EC2

It is better to run the Instana AWS sensor on a Current Generation General Purpose machine that runs Linux®. For example, m5.large instances are ideal.

1. Sign in to Instana and click More > Agents > Instana agent installation > AWS.

2. From the Technology list, select Instana AWS Sensor.

3. From the Run your AWS Agent on list, select Elastic Cloud Compute (EC2) (it's the default)

   Your agent key and location are pre-populated in the script to be used as User Data on EC2. The script looks like the following, except that all the necessary information is provided:

   #!/bin/bash
   
   curl -o setup_agent.sh https://setup.instana.io/agent
   chmod 700 ./setup_agent.sh
   sudo ./setup_agent.sh -y -a <your-agent-key> -m aws -t dynamic -e <location> -s
   

4. Spin up a dedicated EC2 virtual machine and use the script as User Data. For more information on how to user User Data on EC2, see Running Commands on Your Linux Instance at Launch documentation.

5. Copy the configuration.json file for the IAM role. It is the needed IAM role for the EC2 virtual machine that runs the Instana AWS sensor and enables the Instana agent to discover and monitor your AWS resources. Then, create an IAM role with the configuration.json file. For more information, see Creating a role to delegate permissions to an IAM user.

6. For the IAM role to do the AssumedRole action, edit the Trust Relationship. For example,

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

   In case your EC2 instance is in one of China's regions, Trust Relationship is look like:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com.cn"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

   Refer to specifics of the China region monitoring for the additional necessary configuration.

Installation on Fargate on ECS

1. Sign in to Instana and click More -> Agents -> Instana agent installation -> AWS.

2. From the Technology list, select Instana AWS Sensor.

3. From the Run your AWS Agent on list, select Elastic Container Service (ECS)

   The JSON of a Task Definition is pre-generated for you in the Task Definition JSON template. You can use it with the Configure by using JSON functions of the Task Definition user interface.

4. Assign to the new Task Definition an IAM role that contains at least the permissions that are listed under the IAM permissions JSON document.

5. Create an ECS Service with one instance of the newly created Task Definition.

Installation outside your AWS infrastructure

You can also dedicate any agent that runs outside your AWS infrastructure. To achieve this goal, you need to specify the following environment variables within the /opt/instana/agent/bin/setenv file:

• The region that you want to monitor:

  INSTANA_AWS_REGION_CONFIG=
  

• The credentials to access AWS resources. These credentials should belong to a user, which is allowed to access the resources already described in the Amazon Web Services IAM Configuration section.

  AWS_ACCESS_KEY_ID=
  AWS_SECRET_ACCESS_KEY=
  

Proxy configuration

export HTTP_PROXY=
export HTTPS_PROXY=

com.instana.plugin.aws:
  proxy_host: 'example.com' # proxy host name or ip address
  proxy_port: 3128 # proxy port
  proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
  proxy_username: 'username' # OPTIONAL: proxy username
  proxy_password: 'password' # OPTIONAL: proxy password
  tagging: # proxy setup for AWS Tagging API, used for AWS resource monitoring filtering
    proxy_host: 'example.com' # proxy host name or ip address
    proxy_port: 3128 # proxy port
    proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
    proxy_username: 'username' # OPTIONAL: proxy username
    proxy_password: 'password' # OPTIONAL: proxy password

China region specifics

By default, the Instana agent contacts the global STS endpoint from the EC2 instance to gather instance profile credentials. Since this approach doesn't work in CN regions, the STS region endpoint needs to be displayed as an environment variable, which tells the Instana agent to use regional (CN) STS endpoints.

After the Instana agent is installed, specify the following environment variable within the <instana-agent-dir>/bin/setenv:

export AWS_STS_REGIONAL_ENDPOINTS=regional

Enable monitoring for specific services

For each service, you need to add the needed permissions on the individual service pages, linked to from the Monitored Services section. To exclude a service from monitoring, leave out the respective permission.

Alternatively, another way to exclude a service from monitoring is to add the appropriate-enabled_ flag in <agent_install_dir>/etc/instana/configuration.yml as described on the following individual service pages.

Monitoring multiple AWS accounts

The AWS Instana agent supports monitoring services from multiple AWS accounts in the same region. Currently, there are two approaches: using AWS named profiles and by using AWS Security Token Service (STS).

Note: When you configure the Instana agent to monitor multiple AWS accounts, only one of the two available approaches should be used.

[default]
aws_access_key_id = accessKey1
aws_secret_access_key = secretAccessKey1

[profile2]
aws_access_key_id = accessKey2
aws_secret_access_key = secretAccessKey2

[profile3]
aws_access_key_id = accessKey3
aws_secret_access_key = secretAccessKey3

com.instana.plugin.aws:
  profile_names:
    - 'profile2'
    - 'profile3'

com.instana.plugin.aws:
  role_arns:
    - 'arn:aws:iam::<account_2_id>:role/<role_2_name>'
    - 'arn:aws:iam::<account_3_id>:role/<role_3_name>'

Filtering and tagging

Note: To enable filtering, the IAM Permission tag:GetResources should be included in the IAM permissions that are granted to the AWS Agent.

After you enable the monitoring of a service, it is possible to filter, which of its instances Instana will monitor based on AWS tags by modifying the agent configuration file /opt/instana/agent/etc/instana/configuration.yaml of the AWS Agent.

Available options in the configuration file are:

com.instana.plugin.aws:
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is discovered.
  include_tags: ...
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is omitted from discovery.
  exclude_tags: ...
  # Exclude untagged AWS resources by setting the flag to `false`
  include_untagged: ...

Multiple tags can be defined, separated by a comma. Tags should be a key-value pair that is separated by :. When you define tags in both lists (include and exclude), the exclude list has higher priority. If there is no need for filtering services, the configuration should not be defined.

Note: The Instana agent automatically monitors all AWS resources that have no tags that are assigned. To exclude untagged resources from monitoring, set the include_untagged flag to false.

Filtering can also be configured on the service level. In this case, the default configuration is overridden for the particular service. For more details about discovery filtering by specific service, see specific service documentation.

Tag poll rate

To specify how often the AWS sensor polls monitored AWS service to get associated tags, use the tagged-services-poll-rate configuration property. The default is 300 seconds. The sensor uses AWS service client to poll tags.

The poll rate configuration for the tagged resources is:

com.instana.plugin.aws:
  tagged-services-poll-rate: 60 #default 300

Polling interval

The polling interval indicates how often AWS sensor calls the CloudWatch API. It is configurable as a cloudwatch_period in the configuration.yml file. The default value is 300 seconds. It's most common to see monitoring platforms use a value 5 - 10 minutes.

The polling interval is configurable on two levels:

• Agent level for all AWS services monitored by the agent:

  com.instana.plugin.aws:
    cloudwatch_period: 300
  

• Per AWS service:

  com.instana.plugin.aws.rds:
    cloudwatch_period: 300
  

Configuration for single service overrides agent level configuration.

CloudWatch costs

To provide insights into your AWS services, monitoring platforms like Instana must use the CloudWatch API. This API is provided by AWS with consumption-based pricing, and the purpose of this documentation is to provide transparency into how Instana uses the CloudWatch API for users to understand the impact on their AWS bill.

CloudWatch costs' are affected by:

• Number of monitored instances of AWS services
• Number of collected metrics per AWS sensor
• Polling interval (configurable)

Each AWS sensor calls the GetMetricData CloudWatch API request for metrics collection, and if it fails GetMetricStatistics is called as a fallback. Both endpoints incur the same costs per metrics (for clarity: the fallback usage of GetMetricStatistics does not incur extra costs). You can find details for both on the AWS CloudWatch pricing page.

The following table contains roughly estimated daily costs for different AWS sensors when the polling interval is 5 minutes and Amazon charge is $0.01 per 1000 CloudWatch metrics:

Note that the CloudWatch are not used for EC2 and for polling tags, so there are no CloudWatch costs incurred there.

You can reduce CloudWatch costs by increasing the polling interval. The downside to increasing the polling interval is that you reduce granularity metrics from CloudWatch. Each request provides a single metric value, so the difference between a polling interval of 60s and 300s is five metric values or one metric value for a 300s period.

It is up to you to determine whether the default polling interval is right for your business or whether to customize it to strike the balance between cost and quality of insights.