Connect Amazon Web Services

You can connect Amazon Web Services (AWS) account to Cloudability to enable the ingestion of usage data.

To connect your AWS account to Cloudability , perform the following steps:

1. Configure consolidated account credentials
2. Set up access to AWS API

To configure consolidated account credentials, perform the following steps:

1. Create an AWS Cost and Usage report
2. Enable cost allocation tags in the AWS console
3. Generate access credentials in [ Cloudability ]
4. Verify the consolidated account credential

1. From AWS Management Console , select your account name at the top right and select My Billing Dashboard .
2. From the Billing & Cost Management Dashboard page, select Data Exports (under Cost Analysis) .
   1. Select Create .
   2. On the Export Type, choose one of the following.
      
      Select Legacy CUR export

      1. On the Export content, select the following.

         • Select Include resource IDs.

         • Select Refresh automatically.

      2. On the Data export delivery options, select the following.

         • Select Hourly for Report data time granularity.

         • Create new report version for Report versioning.

         • Ensure that the compression type is set to gzip or parquet.

      3. On the Data export storage settings, follow the steps below.

         • Select Configure.

         • Select an existing S3 bucket or create a new one.

         • Select an existing S3 bucket or create a new one.

         • Your Cost and Usage report files will reside in this bucket.

         • Select Create Bucket (or Select Bucket if already created)

         • Enter an S3 path prefix.

         • (Optional) Add any tags if necessary.

         • Click Create Report.

      4. Select either Overwrite existing data export file or Create new data export for file versioning

      Select Standard data export, CUR 2.0

      1. Provide your Export name .
      2. On the Data Table and Content setting, select the following.

         • Select Include resource IDs .

         • Select Refresh automatically .

         • Column Selection – Leave as default (all columns).

           Note:

           Do not select Split cost allocation data.

      3. On the Data export delivery options , select the following.

         • Ensure that the compression type is set to gzip

         • Select Overwrite existing data export file for file versioning.

      4. On the Data export storage settings, follow the steps below.

         • Select Configure .

         • Select an existing S3 bucket or create a new one.

         • Your Cost and Usage report files will reside in this bucket.

         • Select Create Bucket (or Select Bucket if already created)

         • Enter an S3 path prefix .

         • (Optional) Add any tags if necessary.

         • Click Create Report .

         Note:

         Existing customers on AWS CUR, can continue to remain on Legacy CUR and everything would work as is.

3. From the Report content page, do the following:
   • In the Report name - required field, enter a name for your report.
   • Select Include resource IDs .
   • Select Refresh Automatically under Data Refresh settings .
4. From the Data table delivery options page, do the following:
   • Keep the default settings, and
   • Ensure Hourly is selected.
     Ensure Create new report version is selected.
   • Ensure that the compression type us set to gzip .
     
   • Under Data export storage settings :
   • Select Configure .
     The Configure S3 Bucket dialog opens.
   • Select an existing S3 bucket or create a new one.
     Your Cost and Usage report files will reside in this bucket.
   • Select Create Bucket (or Select Bucket if already created) .
   • Enter an S3 path prefix .
   • (Optional) Add any tags if necessary.
   • Click Create Report .

1. From the Billing & Cost Management Dashboard , navigate to Cost Allocation Tags .
2. Select the tags that you want to include.
3. Select Activate .

Create and download the credential template in Cloudability

1. In Cloudability , navigate to Settings > Vendor Credentials > AWS . The Add AWS Account panel opens.

   or

   In Cloudability , navigate to Settings > Vendor Credentials > AWS . Select Add a Credential .

2. Under CREATE IAM ROLE , enter the following:
   • PAYER ACCOUNT ID
   • S3 BUCKET NAME : The bucket that contains your Cost and Usage report
   • COST AND USAGE REPORT NAME
   • COST AND USAGE REPORT PREFIX
   Note:

   To use Cloudability Automation features, “Include Automation permissions” checkbox needs to be checked. If you are not planning to use Cloudability Automation this can be unchecked.

3. In case you prefer Automated credentialing of AWS linked accounts, select the checkbox “Automated credentialing of linked accounts”, otherwise leave it unchecked.
   Note:

   To automate credentialing of your AWS linked accounts, refer to AWS Creating and Managing an Organization .

4. Select Generate Template .

   This option is only visible if you select the pencil icon of an account that has been verified (a green check mark with no background).

5. Select Download .
6. An AWS CFT template will get downloaded which needs to be uploaded into AWS console in the next step.

Upload the credential template to AWS Management Console

1. In AWS Management Console , select the Services list at the top left.

2. Search for 'CloudFormation' and select the result.
3. From the AWS CloudFormation page, select Create stack

   The Create stack page opens. .

4. Under Select template , do the following:
   • Select Upload a template file .
   • Select Choose file and upload the template you downloaded from Cloudability.
   • Select Next .
5. From the Specify stack details page, do the following:
   • Enter a Stack name (for example, something beginning with ' Cloudability ').
   • Verify the populated Parameters .
   • Select Next .
6. Proceed through the Configure stack options page.
7. From the Review page, do the following:
   • Check the acknowledgment box.
   • Select Create stack .

1. In Cloudability , navigate to Settings > Vendor Credentials > AWS .
2. Select Add a Credential . The Add a Credential panel opens.
3. Under CREATE IAM ROLE , enter the following:
   • PAYER ACCOUNT ID
   • S3 BUCKET NAME : the bucket that contains your Cost and Usage report
   • COST AND USAGE REPORT NAME
   • COST AND USAGE REPORT PREFIX
   • AWS SCP ENABLED REGION : In case you have applied AWS service control policy at an AWS region, indicate the region such as us-east-2.
4. In case you prefer Automated credentialing of AWS linked accounts, select the checkbox otherwise leave it unchecked.
   Note:

   To automate credentialing of your AWS linked accounts, refer to AWS Creating and Managing an Organization .

5. At the bottom of the Add a Credential panel, select Verify Credentials .

Download a CloudFormation template

1. In Cloudability , navigate to Settings > Vendor Credentials > AWS .
   Note:

   This page requires admin permissions to access it.

2. Hover your cursor over the icon of the account for which you want to download the template.

   Additional options are displayed.

3. Select the icon to open the Edit a Credential panel.

   The Edit a Credential panel opens.

   Note:

   AWS SCP ENABLED REGION: In case you have applied AWS service control policy at an AWS region. Please indicate the region e.g. us-east-2.

4. Select Generate Template.

5. Select Download .

Regenerate a CloudFormation template

1. In Cloudability , navigate to Settings > Vendor Credentials > AWS .
   Note:

   This page requires admin permissions to access it.

2. Hover your cursor over the icon of the account for which you want to regenerate the CF template.

   Additional options are displayed.

3. Select the icon to open the Edit a Credential panel.
4. In the S3 BUCKET NAME (REQUIRED) field, enter the S3 bucket name of the payer account.
   Note:

   AWS SCP ENABLED REGION: In case you have applied AWS service control policy at an AWS region. Please indicate the region e.g. us-east-2.

   

5. Select Generate Template to download your CF template.

Verify linked account credentials

1. In Cloudability , navigate to Settings > Vendor Credentials > AWS .

   The linked account has a green check box under Advanced Features .

2. Hover your cursor over the icon of the linked account for which you want to verify credentials.

   Additional options are displayed.

3. Select the icon to open Edit a Credential .
4. At the bottom of the panel, select Verify Credentials .
   Note:

   AWS SCP ENABLED REGION: In case you have applied AWS service control policy at an AWS region. Please indicate the region e.g. us-east-2.

Notice the yellow line under the accounts that have existing user credentials. The setup process for migrating these credentials is the same as for adding new credentials, but you need to edit your existing credentials instead of creating new ones.

Regenerate a Cloud Formation template To regenerate the Cloud Formation template to configure your AWS access, do the following:

1. In Cloudability , navigate to Settings > Vendor Credentials > AWS .
   Note:

   The Vendor Credentials page requires admin permissions to access it.

2. Go to stacks service.
3. Find out which stack he has previously run in order to install cloudability's cloud formation template.
4. Select the same stack and click the update button
5. Select ' replace current template ' , and then upload the new CloudFormation template file.
6. On successful updation, re-verify the account.

Here’s how you can update to the latest version for all of your organization's payer and linked accounts.

• Payer accounts (need S3 bucket access): see Configure consolidated account credentials
• Linked accounts (do not need S3 bucket access): see Set up access to AWS API

{ 
					"Version": "2012-10-17", 
					"Statement": [{    
					"Sid": "masterpayerblock",     
					"Effect": "Allow",     
					"Action": ["s3:ListBucket", "s3:GetObject", 
					"s3:GetObjectVersion"],     
					"Resource": [       
					"arn:aws:s3:::add-your-s3-bucket",       
					"arn:aws:s3:::add-your-s3-bucket/*"     
					]  
					}, {     
					"Sid": "linkedaccountblock",     
					"Effect": "Allow",     
					"Action": [       
					"organizations:ListAccounts",       
					"cloudwatch:GetMetricStatistics",       
					"dynamodb:DescribeTable",       
					"dynamodb:ListTables",       
					"ec2:DescribeImages",      
					"ec2:DescribeInstances",       
					"ec2:DescribeRegions",       
					"ec2:DescribeReservedInstances",       
					"ec2:DescribeReservedInstancesModifications",       
					"ec2:DescribeSnapshots",       
					"ec2:DescribeVolumes",       
					"ec2:GetReservedInstancesExchangeQuote",       
					"ecs:DescribeClusters",       
					"ecs:DescribeContainerInstances",       
					"ecs:ListClusters",       
					"ecs:ListContainerInstances",       
					"elasticache:DescribeCacheClusters",       
					"elasticache:DescribeReservedCacheNodes",       
					"elasticache:ListTagsForResource",       
					"elasticmapreduce:DescribeCluster",       
					"elasticmapreduce:ListClusters",       
					"elasticmapreduce:ListInstances",       
					"rds:DescribeDBClusters",       
					"rds:DescribeDBInstances",       
					"rds:DescribeReservedDBInstances",       
					"rds:ListTagsForResource",       
					"redshift:DescribeClusters",       
					"redshift:DescribeReservedNodes",       
					"redshift:DescribeTags",       
					"savingsplans:DescribeSavingsPlans"     
					],     
					"Resource": "*"   
					}]  
					}

{  
					"Version": "2012-10-17",  
					"Statement": [{     
					"Sid": "linkedaccountblock",     
					"Effect": "Allow",       
					"Action": [       
					"cloudwatch:GetMetricStatistics",       
					"dynamodb:DescribeTable",       
					"dynamodb:ListTables",       
					"ec2:DescribeImages",       
					"ec2:DescribeInstances",        
					"ec2:DescribeRegions",       
					"ec2:DescribeReservedInstances",      
					"ec2:DescribeReservedInstancesModifications",       
					"ec2:DescribeSnapshots",       
					"ec2:DescribeVolumes",         
					"ec2:GetReservedInstancesExchangeQuote",       
					"ecs:DescribeClusters",       
					"ecs:DescribeContainerInstances",       
					"ecs:ListClusters",       
					"ecs:ListContainerInstances",       
					"elasticache:DescribeCacheClusters",       
					"elasticache:DescribeReservedCacheNodes",       
					"elasticache:ListTagsForResource",        
					"elasticmapreduce:DescribeCluster",       
					"elasticmapreduce:ListClusters",       
					"elasticmapreduce:ListInstances",       
					"rds:DescribeDBClusters",       
					"rds:DescribeDBInstances",      
					"rds:DescribeReservedDBInstances",       
					"rds:ListTagsForResource",       
					"redshift:DescribeClusters",       
					"redshift:DescribeReservedNodes",       
					"redshift:DescribeTags",       
					"savingsplans:DescribeSavingsPlans"     
					],     
					"Resource": "*"   
					}]  
				}
     

This is the preferred method for memory data collection going forward. Cloudability supports the standard location and naming conventions that the unified agent uses when writing custom metrics to Cloudwatch. These details are:

• metric-name : "mem_used_percent" (for Linux)
• metric-name : "Available Mbytes" (for Windows)
• namespace : CWAgent
• dimensions : InstanceId (it’s important to only add this one dimension)
• unit : percent

						{   
						"metrics": {
						"metrics_collected": {
						"mem": {
						"measurement": [
						"mem_used_percent"
						],
						"resources": [
						"*"
						]
						}
						},  
						"append_dimensions": {
						"InstanceId": "${aws:InstanceId}"
						} 
						}
				} 

The following is an example of a minimal Windows configuration to publish just the memory information:

						{  
						"metrics": { 
						"metrics_collected": { 
						"Memory": { 
						"measurement": [ 
						{"name": "Available Mbytes", "unit":"Megabytes"} 
						], 
						"resources": [ 
						"*" 
						] 
						} 
						},  
						"append_dimensions": { 
						"InstanceId": "${aws:InstanceId}" 
						} 
						} 
						} 
				

The following is a Linux example:

						{   
						"metrics": {
						"metrics_collected": {
						"mem": {
						"measurement": [
						"mem_used_percent"
						],
						"resources": [
						"*"
						]
						}
						},  
						"append_dimensions": {
						"InstanceId": "${aws:InstanceId}"
						} 
						}
				} 

The following is an example of a minimal Linux configuration to publish just the memory and GPU information:

{   
						"metrics": {
						"metrics_collected": {
						"mem": {
						"measurement": [
						"mem_used_percent"
						]       
						},
						"nvidia_gpu": {
						"measurement": [
						"utilization_gpu",
						"utilization_memory"
						]
						}    
						},  
						"append_dimensions": {
						"InstanceId": "${aws:InstanceId}"
						} 
						}
				} 

Your VM should have the following:

• GPU enabled
• Nvidia drivers installed
• Python 2.7 or higher installed