AWS Quick Start Guide

About

SevOne's AWS plugin allows you to collect devices, metadata, and metrics from the AWS environment right out-of-the-box. The plugin makes use of AWS CloudWatch metric streams for overtime data as well as for API calls for metadata enrichment. The AWS plugin allows you to monitor multiple different AWS accounts and regions.

Device Creation

SevOne NMS is configured, by default, to regularly call AWS APIs to retrieve information about the devices and then, automatically create devices for each AWS resource.

Configure Web Proxy for AWS Plugin

Important: Please refer to section Appendix A: Web Proxy Setup to set up the proxy using the environment variables in /etc/sevone/nms-collectors.yaml deployment file.

Required Infrastructure in AWS

To monitor an AWS account, the following necessary infrastructure in AWS must be configured

IAM Users
IAM Collector Role
SQS Queue
Metric Stream
Firehose
S3 Bucket

This infrastructure is used to collect metrics for your AWS environment and make them available to SevOne NMS. Ensure that this is configured before continuing with the AWS plugin.

Note: AWS Infrastructure can be set up in two ways.

Terraform (recommended) - for details, please continue to section AWS User Setup.
AWS management console - for details, please refer to section Appendix B: Deploy AWS Resources using AWS Console to learn how to setup via AWS management console.

AWS User Setup

To maintain a least privilege posture when setting up the required infrastructure in AWS and running the collector, the following is recommended.

Infrastructure User - user deploying the infrastructure.
Collector User - user used by the collector.

Infrastructure User Setup

Create a new IAM Policy with the following permissions.


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "StatementIAM",
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:UpdateAssumeRolePolicy",
        "iam:ListAttachedRolePolicies",
        "iam:CreateRole",
        "iam:ListRolePolicies",
        "iam:GetRolePolicy",
        "iam:PutRolePolicy",
        "ec2:DescribeRegions
      ],
      "Resource":[
        "*"
      ]
    }
  ]
}

Create a new user and attach it with the IAM Policy created above.

Note: For additional details about creating IAM users in your AWS account, please refer to https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html.

Collector User Setup

Create a second new IAM User. There is no need to attach a policy to this user.

The collector user simply needs to exist and can have an empty permissions set. It will assume a role with the required permissions created by Terraform in the step below.

Note: For additional details about creating IAM users in your AWS account, please refer to https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html.

Note: The ARN of this Collector User is required for sections in this guide below.

Access Key Credentials

For the sections in this guide below, you will need to create an Access Key credential for the Collector User and retain its Access Key ID and Secret Key. It is important to copy the Secret Key immediately after creating the credential, as it will be permanently hidden when the page is left.

An Access Key credential is also required for the Infrastructure User if deploying infrastructure with Terraform. In that case, you will need the Access Key ID and Secret Key for this credential.

Note: For additional details on creating access keys for IAM users, please refer to https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html.

Deploy with Terraform

Terraform is the recommended way to configure the AWS resources. This allows for the quickest startup time while ensuring that resources are configured exactly as intended. To deploy and configure the required resources, a set of Terraform files can be found in /config/aws/terraform directory. In order to run the Terraform files to create the resources, a role has been defined to maintain a least privileged posture. Let's refer to this role as Infrastructure Role.

Create an Infrastructure Role for Terraform to use

Note: The Infrastructure Role can be created in two ways.

Terraform (recommended) – for details, continue on with this section.
AWS management console – for details, please refer to Appendix C: Create an Infrastructure Role via AWS Console for Terraform to use

Once an Infrastructure Role has been created, continue with section Run Terraform to Deploy AWS Resources below.

Create Infrastructure Role for Terraform

Using ssh, log in to SevOne NMS appliance as support user.
```
$ ssh support@<NMS appliance>
```

Open a shell within the NMS container.

$ sudo podman exec -it nms-nms-nms /bin/bash

Change directory to /config/aws/terraform/envs/infrastructure_role.

Update the following values in terraform.tfvars.

Important: Variable prefix must be,

between 1 and 20 characters
contain only lowercase letters, digits, or hyphens
start and end with letters or digits


account_id         = <YOUR AWS ACCOUNT NUMBER>
prefix             = <PREFIX TO UNIQUELY IDENTIFY RESOURCES>

Empty 'terraform.tfvars' file published with examples in comments


## Your 12 digit AWS account number
# account_id              = 012345678901

## Prefix to uniquely identify resources that are created in AWS
## - Must be all lowercase due to S3 bucket naming restrictions
# prefix                  = "sevone"

Set the following environment variables with Access Key ID and Secret Key for the Infrastructure User that was created in section Access Key Credentials. For more information about creating access keys for IAM users, please refer to https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
```
export AWS_ACCESS_KEY_ID="mykey"
export AWS_SECRET_ACCESS_KEY="mysecret"
export AWS_REGION="us-east-1"
```
Apply the terraform files.
```
terraform init
terraform plan
terraform apply
```
Important: Please make note of output value, infrastructure_role_arn, as it will be required in section Run Terraform to deploy AWS resources below.

Run Terraform to deploy AWS resources

Change directory to /config/aws/terraform/envs/collector_infrastructure.
```
$ cd /config/aws/terraform/envs/collector_infrastructure
```

Update the following values in terraform.tfvars.

Empty 'terraform.tfvars' file published with examples in comments


## Your 12 digit AWS account number
# account_id              = 012345678901
 
## AWS IAM User ARN that will be used to run the collector
# collector_user_arn      = "arn:aws:iam::012345678901:user/person@company.com"
 
## AWS IAM Role ARN that will be used to install infrastructure
## - This can be found in the output of applying the `infrastructure_role` Terraform config
# infrastructure_role_arn = "arn:aws:iam::012345678901:role/sevone_infrastructure_role"
 
## List of regions that you want to monitor
# regions                 = ["us-east-1", "us-west-1"]
 
## Prefix to uniquely identify resources that are created in AWS
## - Must be all lowercase due to S3 bucket naming restrictions
# prefix                  = "sevone"

Important: Variable prefix must be the same prefix that was used when creating the Infrastructure Role.

Important: Resources must be created in every region that you want to monitor. When adding more than one region to create the resources in, you can add it in terraform.tfvars file as,

Example


regions = ["us-east-1", "eu-central-1"]

Apply the terraform files.


cd gen
terraform init
terraform plan
terraform apply
cd ..
terraform init
terraform plan
terraform apply

Details from the following output will be required to create a device in SevOne NMS.
1. account_id
2. collector_role_arn
3. sqs_queue
4. regions

Enable AWS Plugin

Execute the following steps to monitor an AWS account. This will automatically create devices and collect metrics for various AWS resources within selected regions for that account.

To access the Device Manager from the navigation bar, click the Devices menu and select Device Manager.
Either add a device with the AWS plugin or edit an existing device to enable the AWS plugin.
- Click Add Device to display the New Device page.
- Click the wrench icon under the Actions column to display the Edit Device page.
Click the plugin drop-down. By default, it is set to SNMP. Select AWS.
Select the Enable AWS Integration check box.
In the Account ID field, enter the ID of the account you want to monitor and collect data from.
In the Access Key ID field, enter the Access Key ID previously created in Access Key Credentials for the Collector User.
In the Secret Access Key field, enter the Secret Key previously created in Access Key Credentials for the Collector User.
In the SQS Queue field, enter the SQS Queue name that the collector will listen to for metric stream events. For additional details, please refer to section Required Infrastructure in AWS.
In the Collector Role ARN field, enter the ARN of the IAM role that the collector will assume. For additional details, please refer to section Required Infrastructure in AWS.
Select / deselect the column headers or AWS region codes you want to collect data from.
1. Select or deselect a column header (for example, North America) to enable or disable collection for all regions underneath that header.
2. Select or deselect an AWS region code (for example, us-east-1) to enable or disable collection for that region.
Click Save As New to save the current changes as a New Device, or click Save to confirm the changes in the Edit Device page.
When the desired changes have been saved, click the Cancel button to return to the Device Manager page.
Important: Once the device is created, collection of AWS resources starts on SevOne NMS.

Create TopN View in SevOne NMS to view AWS Resources

SevOne NMS contains a package with AWS-specific TopN views. To import, execute the following commands.


$ ssh support@<NMS appliance> 

$ sudo podman exec -it nms-nms-nms SevOne-import --file /config/aws/topn.spk

Important: The import will fail unless the AWS collector has run successfully at least once.

OOTB Reports

The following out-of-the-box (OOTB) reports are available as part of SevOne Data Insight 6.5 and above.

AWS Direct Connect - report showing AWS Direct Connect inventory and network performance.
AWS EBS Volumes - report showing AWS EBS Volume performance.
AWS EC2 Report - report showing AWS EC2 inventory, CPU, disk, and network performance.
AWS NAT Gateway - report showing AWS NAT Gateway inventory, throughput and connection statistics.
AWS Network Load Balancers - report showing AWS Network Load Balancer statistics.
AWS S3 Report - report showing AWS S3 inventory and bucket statistics.
AWS Site to Site VPN - report showing AWS Site to Site VPN metrics.
AWS Transit Gateways - report showing AWS Transit Gateway inventory, network traffic, and drops.

Update AWS Infrastructure

SevOne NMS v7.0.0 supports more AWS resources for monitoring than v6.6.x and consolidates the required AWS permissions to maintain a least-privilege posture. For this reason, if upgrading from SevOne NMS 6.8 or a prior version to SevOne NMS 7.0, you are required to reapply Terraform to update the collector’s infrastructure.

SevOne NMS v7.0.0 also makes multiple changes to the Terraform input variables. Any previous terraform.tfvars files have been renamed to terraform-v6.8.0.tfvars. They can be used as references to set the new versions of terraform.tfvars.

Important: SevOne NMS v7.0.0 is incompatible with previously generated Terraform resources if the prefix variable differs between /config/aws/terraform/envs/infrastructure_role/terraform-v6.8.0.tfvars and /config/aws/terraform/envs/collector_infrastructure/terraform-v6.8.0.tfvars. If the prefixes are the same, skip to section Update Infrastructure Role below. If the prefixes are different, follow the steps below to destroy the existing AWS resources.

Destroy Previously Created AWS Resources

Execute the following steps to destroy the previously created AWS resources.

Using ssh, log in to SevOne NMS appliance as support user.
```
$ ssh support@<NMS appliance>
```

Open a shell within the NMS container.


$ sudo podman exec -it nms-nms-nms /bin/bash

Change directory to /config/aws/terraform/envs/collector_infrastructure.
```
$ cd /config/aws/terraform/envs/collector_infrastructure
```
Destroy the Terraform state.
```
$ terraform destroy
```

Once the resources have been destroyed, continue to the next section.

Update Infrastructure Role

If not already, use ssh to log in to SevOne NMS appliance as support user and open a shell within the NMS container.
```
$ ssh support@<NMS appliance>

$ sudo podman exec -it nms-nms-nms /bin/bash
```
Change directory to /config/aws/terraform/envs/infrastructure_role.
Compare the new terraform.tfvars to the backup terraform-v6.8.0.tfvars, if it exists.
Using a text editor of your choice, edit terraform.tfvars by uncommenting the variable assignments and setting the variables to the correct values.

Apply the terraform files.


terraform init
terraform plan
terraform apply

Important: Please make note of output value, infrastructure_role_arn, as it will be required in section Run Terraform to deploy AWS resources below.

Run Terraform to deploy AWS resources

Change directory to /config/aws/terraform/envs/collector_infrastructure.

Regenerate the terraform files.


cd gen
terraform init
terraform plan
terraform apply
cd ..

Compare the new terraform.tfvars to the backup terraform-v6.8.0.tfvars, if it exists.
Using a text editor of your choice, edit terraform.tfvars by uncommenting the variable assignments and setting the variables to the correct values.

Important: Variable prefix must be the same prefix that was used for the Infrastructure Role.

Apply the terraform files.


terraform init
terraform plan
terraform apply

Appendices

Appendix A: Web Proxy Setup

If you are using Web Proxies for connecting to the AWS environment, you may need to set the web proxy. The proxy URL can be in one of the following two formats.


http://<user>:<password>@<domain name or IP address>:<port>

or

https://<user>:<password>@<domain name or IP address>:<port>

where, <user> and <password> are URL encoded strings.

These URLs are defined using the environment variables. For example,


HTTP_PROXY=http://user1:password@proxy.corp.com:8080

or 

HTTPS_PROXY=https://user1:password@proxy.corp.com:8080


http://<domain name or IP address>:<port>

or

https://<domain name or IP address>:<port>

if there is no username and password associated with the proxy.

These URLs are defined using the environment variables. For example,


HTTP_PROXY=http://proxy.corp.com:8080

or 

HTTPS_PROXY=https://proxy.corp.com:8080

As the AWS collector is running within a pod as a container, the environment variables must be set for that container. Execute the following steps.

Using ssh, log in to SevOne NMS appliance as support user.
```
$ ssh support@<NMS appliance>
```
Using a text editor of your choice, edit /etc/sevone/nms-collectors.yaml file.
```
$ sudo vi /etc/sevone/nms-collectors.yaml
```
In this file, look for section spec.containers and find the container that is associated with the AWS collector; it should be referred to as aws-collector.

In this section, add the environment variables under env block as required.


"env":
  - "name": "SSL_CERT_DIR"
    "value": "/secrets/nginx"
  - "name": "HTTP_PROXY"
    "value": "http://proxy.corp.com:8080"
  - "name": "HTTPS_PROXY"
    "value": "http://proxy.corp.com:8080"
"image": "docker.s1artrtp1.s1.devit.ibm.com/cloud-monitoring/aws-nms-collector:3e3f2279"
"imagePullPolicy": "IfNotPresent"
"name": "aws-collector"

Restart nms-collectors service.


$ sudo systemctl restart nms-collectors;

Appendix B: Deploy AWS Resources using AWS Console

AWS Console allows you to create the required AWS resources without the use of Terraform. While it is recommended to use Terraform, this is a viable option when that is not possible.

Create a Metric Stream

Navigate to CloudWatch > Metric Streams.
Click Create metric stream to launch the wizard.
Select Select metrics.
Select Include.
Select the following namespaces and select Use: "AWS/<namespace> = All metrics names" to get all metrics for the selected namespaces.
1. AWS/EC2
2. AWS/S3
3. AWS/NATGateway
4. AWS/TransitGateway
5. AWS/DX
6. AWS/EBS
7. AWS/NetworkELB
8. AWS/VPN
Name the metric stream.
Note: You will need the name of the S3 bucket within the Resources will be added to your account section for the next section.
Click Create metric stream to complete the configuration.

Set up S3 Event Notifications

In https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html, follow Steps 1 and Step 3, Option A. to send all object create events to the SQS queue.

Note: You will need the name of the SQS queue you created for the next section and to enable the AWS Plugin.

Create a Collector Role

Create a IAM role for the AWS plugin to use with the following policies.

Note: You will need the ARN of this role to enable the AWS Plugin.

Policy 'aws_collector_directconnect_policy'


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "directconnect:DescribeLocations",
                "directconnect:DescribeConnections"
            ],
            "Resource": "*"
        }
    ]
}

Policy 'sevone_collector_cloudwatch_policy'


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "cloudwatch:GetMetricData",
            "Resource": "*"
        }
    ]
}

Policy 'sevone_collector_ec2_policy'


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeNatGateways", 
                "ec2:DescribeTransitGateways",
                "ec2:DescribeVolumes", 
                "ec2:DescribeVolumeStatus", 
                "ec2:DescribeVpnConnections"
            ],
            "Resource": "*"
        }
    ]
}

Policy 'sevone_collector_elasticloadbalancing_policy'


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:DescribeLoadBalancers"
            ],
            "Resource": "*"
        }
    ]
}

Policy 'sevone_collector_s3_policy'

Note: Replace the BUCKET_NAME with the name of your S3 bucket from Create a Metric Stream .


{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Sid": "", 
            "Effect": "Allow", 
            "Action": [ 
                "s3:ListAllMyBuckets" 
            ], 
            "Resource": "*" 
        }, 

        { 
            "Sid": "", 
            "Effect": "Allow", 
            "Action": [ 
                "s3:GetObject" 
            ], 
            "Resource": "arn:aws:s3:::BUCKET_NAME*" 
        }, 
    ] 
}

Policy 'sevone_collector_sqs_policy'

Note: Replace the QUEUE_NAME with the name of your SQS queue from the Set up S3 Event Notifications section, and replace the ACCOUNT_ID with the SQS queue’s AWS account ID.


{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Sid": "", 
            "Effect": "Allow", 
            "Action": [ 
                "sqs:ReceiveMessage", 
                "sqs:GetQueueUrl", 
                "sqs:DeleteMessage" 
            ], 
            "Resource": "arn:aws:sqs:*:ACCOUNT_ID:QUEUE_NAME" 
        } 
    ] 
}

Finally, you will need to allow the Collector User to assume the newly created role.

Navigate to IAM > Roles.
Search for the collector role that you recently created
Click the role.
Note: If you have not already copied the ARN of this role, you can do so now. You will need it to enable the AWS Plugin.
Click the Trust Relationships tab.
Ensure that the Collector User has permission to assume the role.
- If the Collector User does not have permission, you can add a new statement to the trust policy with the Collector User ARN like the one below by clicking the Edit trust policy button.

Note: Replace the COLLECTOR_USER_ARN with the Collector User ARN.


{ 
  "Version": "2012-10-17", 
  "Statement": [
     { 
       "Sid": "", 
	   "Effect": "Allow", 
   	"Principal": { 
		       "AWS": "COLLECTOR_USER_ARN" 
		       }, 
   	"Action": "sts:AssumeRole" 
        } 
    ] 
}

Once all the steps have been completed, you may Enable AWS Plugin.

Appendix C: Create an Infrastructure Role via AWS Console for Terraform to use

Create a Policy

To create a role for Terraform to use, you must first define the policy that will be added to the role.

Navigate to IAM > Policies.
Click the Create policy button to launch the wizard.
Switch the Policy editor to JSON.

Paste in the following policy definition.

Important: Variable prefix must be,

between 1 and 20 characters
contain only lowercase letters, digits, or hyphens
start and end with letters or digits

This prefix is used for the Terraform and appended to all resources created for the AWS Plugin when the Terraform runs.

Note: You will need to replace the instances of PREFIX in the policy statements below with the prefix you picked out, and then replace the instances of ACCOUNT_ID with the AWS account ID for the account that the resources will be created in.


{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Action": [ 
                "cloudwatch:TagResource", 
                "cloudwatch:ListTagsForResource" 
            ], 
            "Effect": "Allow", 
            "Resource": "arn:aws:cloudwatch:*:ACCOUNT_ID:*PREFIX*", 
            "Sid": "" 
        }, 
        { 
            "Action": [ 
                "firehose:ListTagsForDeliveryStream", 
                "firehose:DescribeDeliveryStream", 
                "firehose:DeleteDeliveryStream", 
                "firehose:CreateDeliveryStream" 
            ], 
            "Effect": "Allow", 
            "Resource": "arn:aws:firehose:*:ACCOUNT_ID:deliverystream/PREFIX*", 
            "Sid": "" 
        }, 
        { 
            "Action": [ 
                "iam:TagRole", 
                "iam:PutRolePolicy", 
                "iam:PassRole", 
                "iam:ListRolePolicies", 
                "iam:ListInstanceProfilesForRole", 
                "iam:ListAttachedRolePolicies", 
                "iam:GetRolePolicy", 
                "iam:GetRole", 
                "iam:DeleteRolePolicy", 
                "iam:DeleteRole", 
                "iam:CreateRole" 
            ], 
            "Effect": "Allow", 
            "Resource": " arn:aws:iam::ACCOUNT_ID:role/PREFIX*", 
            "Sid": "" 
        }, 
        { 
            "Action": [ 
                "cloudwatch:StopMetricStreams", 
                "cloudwatch:StartMetricStreams", 
                "cloudwatch:PutMetricStream", 
                "cloudwatch:ListMetricStreams", 
                "cloudwatch:GetMetricStream", 
                "cloudwatch:DeleteMetricStream" 
            ], 
            "Effect": "Allow", 
            "Resource": " arn:aws:cloudwatch:*:ACCOUNT_ID:*PREFIX*", 
            "Sid": "" 
        }, 
        { 
            "Action": [ 
                "s3:PutObject", 
                "s3:PutLifecycleConfiguration", 
                "s3:PutBucketTagging", 
                "s3:PutBucketPublicAccessBlock", 
                "s3:PutBucketNotification", 
                "s3:PutBucketAcl",
                "s3:ListBucket", 
                "s3:ListAllMyBuckets", 
                "s3:GetReplicationConfiguration", 
                "s3:GetLifecycleConfiguration", 
                "s3:GetEncryptionConfiguration", 
                "s3:GetBucketWebsite", 
                "s3:GetBucketVersioning", 
                "s3:GetBucketTagging", 
                "s3:GetBucketRequestPayment",
                "s3:GetBucketPublicAccessBlock", 
                "s3:GetBucketPolicyStatus", 
                "s3:GetBucketPolicy", 
                "s3:GetBucketObjectLockConfiguration", 
                "s3:GetBucketNotification", 
                "s3:GetBucketLogging", 
                "s3:GetBucketLocation", 
                "s3:GetBucketCORS", 
                "s3:GetBucketAcl",
                "s3:GetAccelerateConfiguration", 
                "s3:DeleteBucket",
                "s3:CreateBucket" 
            ],
            "Effect": "Allow", 
            "Resource": "arn:aws:s3:::*PREFIX*", 
            "Sid": "" 
        }, 
        { 
            "Action": [ 
                "sqs:TagQueue", 
                "sqs:SetQueueAttributes", 
                "sqs:ListQueues", 
                "sqs:ListQueueTags", 
                "sqs:GetQueueUrl", 
                "sqs:GetQueueAttributes", 
                "sqs:DeleteQueue", 
                "sqs:CreateQueue" 
            ], 
            "Effect": "Allow", 
            "Resource": "arn:aws:sqs:*:ACCOUNT_ID:*PREFIX*", 
            "Sid": "" 
        } 
    ] 
}

awsSpecifyPermissions

Click Next.
Under the Policy details, enter a Policy name.
Click Create policy.

Create a Role

Once the policy has been created, you can now create the IAM role.

Navigate to IAM > Roles.
Click the Create role button to launch the wizard.
Under Trusted entity type, select AWS account.
Under An AWS account, select This account.
Click Next.
Under Permissions policies search for the name of the policy you created in the previous step.
Click the check box beside the policy you created in the previous step.
Click Next.
Under the Role details, enter a Role name
Click Create role.

Note: Please make note of the new role’s ARN, as it will be required in the Run Terraform to deploy AWS resources.