Configuring Amazon Web Services (AWS) integrations

Edit online

To collect data from Kubernetes and Virtual Machines, install the Amazon Web Services (AWS) integration.

Gathering data

This integration collects the following type of information:

Verifying prerequisites

Installing

  1. Verify the public GA image path of the integration for Amazon Web Services (for example: cp.icr.io/cp/cp4waiops/ibm-mm-cdc-conn:4.3-latest). Run the podman images command.
  2. Log in as a root user on a Linux® host machine that has network access to Amazon Web Services. The Amazon Web Services integration pulls information from Amazon Web Services by using a remote TCP connection.
  3. To log in before you download the public image of integration for Amazon Web Services, run the podman login <cdc-mm ga-image-path> command.
    podman login cp.icr.io/cp/cp4waiops/ibm-mm-cdc-conn:4.3-latest
    For more information about the username and password to use, see step 5 in the Preparing your cluster topic.
  4. Create a directory to store the integration-related configuration file and bash script. 
    mkdir -p /root/cdc
cd /root/cdc
  5. To define connection information to the Metric Manager API, create a Metric Manager backend configuration file with the name: com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg. 
    # Metric Manager configuration file
# Metric Manager's URL
host=http://<metricManagerHost>.ibm.com

# Metric Manager's port
port=18080

# Metric Manager's username for REST API
username=system

# Metric Manager's password for REST API 
# password has been mask ****
password=**********

# Metric Manager's tenant id
tenant_id=APM
  6. Create the configuration-aws.yaml sensor configuration file. Define the Amazon Web Services endpoint, API key, and the metric entities information as in the following example configuration-aws.yaml file for a Amazon Web Services sensor. 
    com.instana.plugin.awsunf:
  enabled: true
  region: us-east-1
  aws_access_key_id: AKIAIOSFODNN7EXAMPLE
  aws_secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  poll_rate: 60
  metrics:
    enabled: true
    poll_rate: 60
    entities:
      entity1:
        poll_rate: 60
        namespace: xxxxxx
        metrics:
        - name: metric1
          filter: regex
        - name: metric2
          filter: regex
      entity2:
        poll_rate: 60
        queries:
        - id: xxxxx
          label: xxxxx
          expression: xxxxx
          return_data: xxxxx
        - id: xxxxx
          label: xxxxx
          metric_stat:
            metric:
              namespace: xxxxx
              metric_name: xxxxx
              dimensions:
              - name: xxxxx
                value: xxxxx
            stat: xxxxx
          return_data: xxxxx
  7. If you want to use vault, complete the following steps:
    1. Add the app secret information to the vault server.
    2. Mount the vault PEM file in the image.
    3. Run the bootstrap script to start the docker image.
    4. Run the docker ps command to check the container ID and access to the container by the docker exec -ti <container_id> bash command.
    5. In the container, add the vault IP address into the /etc/hosts file.
      9.x.x.159 Vault
    6. Check the connection to the vault server.
      ping vault
      Note: If ping isn't available, run the dnf install iputils -y command.
    7. Go to the path where the Amazon Web Services configuration YAML file is located.
    8. Edit the configuration.yaml to add the vault configuration.
      com.instana.configuration.integration.vault:
  connection_url: 'https://Vault:8200' # Mapping through hosts file since PEM ca cert does not contain hostname
  token: '<vault_token>'
  path_to_pem_file: '/root/agentdev/agent-installer/instana-agent/etc/instana/vault-ca.pem'
  secret_refresh_rate: 24
  kv_version: 2
    9. Modify the sensor configuration to use the vault type in the configuration-aws.yaml file.
    10. Restart the integration and check whether the Amazon Web Services sensor can connect and receive metrics.
  8. Create a bash script with execution permission, as in the following example bash script for a Amazon Web Services sensor. 
    podman run \
  -itd \
  --name instana-agent-metric-manager-ga \
  --volume /var/run:/var/run \
  --volume /run:/run \
  --volume /dev:/dev:ro \
  --volume /sys:/sys:ro \
  --volume /var/log:/var/log \
  --volume <cdc-root-path>/configuration-aws.yaml:/opt/instana/agent/etc/instana/configuration-aws.yaml \
  --mount type=bind,source=<cdc-root-path>/com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg,target=/opt/instana/agent/etc/instana/com.instana.cdc.metricmanager.sender.MetricManagerBackend-1.cfg \
  --privileged \
  --net=host \
  --pid=host \
  --env INSTANA_PRODUCT_NAME="metric-manager" \
  --env AGENT_MAX_MEM=6G \
  <IBM-CDC-Public-GA-Image-Path>/ibm-mm-cdc-conn:4.5-latest
  9. Run the bash script to set up and configure the instance for the integration.
Note: If you don't want to monitor everything in your Amazon Web Services integration, or if you have many management zones, you can specify the zones that you do want to monitor. Specify the zones to be monitored in your configuration file. If you have many zones, you might encounter an Out of Memory error when the integration reports on every one of your Amazon Web Services zones. You can set the zones when you configure your integration by adding values to the zone field of your configuration. For more information about zones, or if you want to make other changes to the default configuration, see the Configuring section. For example, if you monitor approximately 200 hosts, you might not need to specify zones in your configuration. Conversely, if you monitor 5000 hosts that are grouped into hundreds of management zones, it's likely worthwhile to narrow them down.
The Amazon Web Services integration is installed and set up on the Linux host.

Verifying the installation

  1. Verify whether the integration instance is up and running.
    $ podman ps
CONTAINER ID   IMAGE                                                                                                                                 COMMAND                  CREATED        STATUS        PORTS     NAMES
3c75a6d23ca8   cp.icr.io/cp/cp4waiops/ibm-mm-cdc-conn:4.3-latest   "/usr/local/bin/tini…"   2 weeks ago  Up 2 weeks ago             instana-agent-metric-manager-ga
  2. Check the logs to confirm that Amazon Web Services metrics are forwarded to Metric Manager.
    $ podman logs -f <container_id>
    Example logs, which show that the metrics are forwarded:
    2023-10-05T12:12:09.543+00:00 | INFO  | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | MetricManager : MetricManagerConfig{Host=http://test.ibm.com, Port=18080, Username=system 
2023-10-05T12:12:09.544+00:00 | INFO  | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | MetricManager : metricManagerURL : http://test.ibm.com:18080/metrics/api/1.0/metrics
2023-10-05T12:12:10.026+00:00 | INFO  | tana-agent-scheduler-thread-13-2 | icManagerBackend | cdc-metricmanager-sender - 1.0.0 | Successfully sent payload to Metric Manager
2023-10-05T12:12:10.026+00:00 | WARN  | tana-agent-scheduler-thread-13-2 | SensorTicker     | com.instana.agent - 1.1.697 | Sending metrics with 1260411 chars took 255815 ms

Configuring

You can edit the configuration-.yaml file to further configure your Amazon Web Services integration.
  1. Go to your configuration-.yaml file on the Linux host machine where you installed your Amazon Web Services integration.
  2. Open the file with your preferred text editor and find the Amazon Web Services section. By default, it looks like the following example but the optional fields are empty.
    com.instana.plugin.awsunf
  enabled: true                                  # Required
  region:                <aws-region-name>       # Required
  aws_access_key_id:     <aws-access-key-id>     # Required
  aws_secret_access_key: <aws-secret-access-key> # Required
  poll_rate: 60                                  # Required
  metrics:
    enabled: true                                # Required
    poll_rate: 60                                # Optional
    entities:                                    # Optional
  3. Edit the values that you want to change, and save the file. The following table lists the variables that can be configured for Amazon Web Services.
    Variable Description Type Default value Required or optional
    enabled Set to true or false to enable or disable the integration. Boolean true Optional
    region AWS Region name. String N/A Required
    aws_access_key_id AWS access key id. String N/A Required
    aws_secret_access_key AWS secret access key. String N/A Required
    poll_rate The number of seconds between queries. Number 60 Optional
    metrics: enabled Set to true or false to enable or disable the metrics integration. Boolean true Optional
    metrics: poll_rate The number of seconds between queries, to be applied if an entity type does not specify its own rate . Number 60 Optional
    metrics: entities A list of entities for metric integration. String N/A Required