Cloudability Connect Google Cloud

Edit online

Analyze your Google Cloud Platform (GCP) cloud spend by enabling GCP data ingestion into Cloudability. The following instructions show you how to configure your credentials at the account level and the project level.

Note:

The Billing Query project will be used to retrieve data from the BQ table for billing queries, as well as from the custom_pricing table for custom pricing information.

Before you start

Cloudability follows cloud best practices to access data in your cloud environment.

Confirm IAM permissions

Cloudability uses your billing Table ID to generate a script that contains gcloud IAM commands which do the following:
  • Create a custom role with permissions specific to reading billing data.
  • Add our Service Account as a member with this custom role.

Both of these actions are taken at the billing-project level.

A similar script is generated when enabling Advanced Features for your GCP projects. For more information, see Billing script .

To be able to successfully run our script from within your cloud console, your gcloud user needs to be granted the following IAM permissions at the project level:
  • iam.roles.create
  • resourcemanager.projects.setIamPolicy
  • resourcemanager.projects.getIamPolicy

If you are adding a billing account in Cloudability , then you must be granted these permissions on the corresponding billing project.

If you are enabling Cloudability 's Advanced Features for a specific project, you must also be granted these permissions on that project.

Check the IAM section of the appropriate project to determine whether you have these permissions.

Enable the Google Cloud Resource Manager API

Cloudability uses Google's Cloud Resource Manager API to test whether the necessary permissions have been granted to support the available features. Google's Cloud Resource Manager API provides many advantages at zero cost to you. You can read more about it at Resource Manager .

A list of APIs made available via the Resource Manager can be found at Cloud Resource Manager API .

Cloudability relies on the projects.testIamPermissions API to test whether specific permissions have been granted to support billing and advanced features. For more information, see Method: projects.testIamPermissions .
  1. Under APIs & Services > Library , search for 'Cloud Resource Manager API'.
  2. From the Cloud Resource Manager API page, select ENABLE .
  3. Make sure that API enabled is ticked.

Find your Table ID

The complete Table ID for your billing account is required to begin the process. You have a Table ID if you currently export your billing data to BigQuery. You can find it looking in BigQuery, making sure you are in the Project where the Billing Data is located. If you expand your Dataset where the data is being exported to, and click Details, the Table ID can be copied for entry into Cloudability.

If not, you are required to set this up.

Working in the Google Console, type Billing in the search bar and when the results show select Billing accounts.

Select Billing export.

Select Edit Settings (based on the Type of Export you are putting in place.

Select your Project where your Billing Data will be held and then click in Data Set and select Create a New Data Set.

Add a Dataset ID and click Create Data Set.

Optionally change the Region OR use multi-region.

Select Save to complete the process.

Note:

GCP Standard usage cost table name should be in this format gcp_billing_export_v1_<BILLING_ACCOUNT_ID> .

Note:

GCP Detailed usage cost table name should be in this format gcp_billing_export_resource_v1_<BILLING_ACCOUNT_ID> .

Customer’s Standard usage / Detailed usage cost table should have the below partitioning detail as mentioned in the below image.

Note:

Google has a tutorial: Export Billing Data to BigQuery .

Note:

Cloudability supports both Standard usage cost table and Detailed usage cost table.

After you have enabled the BigQuery export, it might take a few hours for the billing table to be created, after which you will be able to locate the Table ID.

You can find the billing Table ID by navigating to the project that contains the BigQuery export of your billing data, under Table info > Table ID .

Billing script

Cloudability 's Service Account must have access to read your billing data to ingest it.

You must specify a project and dataset to which the billing data will be exported, when you enable the BigQuery export for a Billing Account . A table is automatically created for you within the specified dataset. This table is referred to as the billing table and our Service Account must be able to read data from this table.

The script performs two steps: it first sets up a custom role within the billing project and then adds our Service Account as a member of the project, binding the custom role. This ensures that our Service Account can read data only from BigQuery tables within a billing project. We do not access BigQuery data in non-billing projects.

1. Custom role setup

The script first creates a custom role within the billing project. Following the Principle of Least Privilege, the script attaches the minimum permissions necessary to read billing data to the custom role, i.e.:
  • bigquery.jobs.create

  • bigquery.tables.getData 

# Example: Create billing custom role for my-billing-project-123 # Billing project ID is my-billing-project-123 gcloud iam roles create CloudabilityRole_Billing \ --project \ my-billing-project-123 \ --title \ "Cloudability Billing Role" \ --description \ "Allows Cloudability access to billing account data" \ --permissions \ bigquery.jobs.create,bigquery.tables.getData \ --stage=GA

2. Add Service Account as Member and Bind Custom role

Once the custom role has been created, the script adds Cloudability 's Service Account as a member of the billing project and binds the custom role to it. 

# Example: Add Cloudability's Service Account as member of my-billing- project-123 # Billing project ID is my-billing-project-123 gcloud projects add-iam-policy-binding my-billing-project-123 \ --member serviceAccount:billing-data-service-acct@cloudability- credentials.iam.gserviceaccount.com \ --role 'projects/my-billing-project- 123/roles/CloudabilityRole_Billing'

3. GCS Bucket permissions

If you are opting for GCP standard billing with GCS bucket or GCP detailed billing, a GCP GCS bucket has to be configured. Cloudability will export data to this GCS bucket temporarily from the configured GCP BigQuery table to ingest this data to Cloudability. Once the data ingestion is complete Cloudability will delete the data from the bucket.

The following permissions are required:
  • bigquery.tables.export (export table data out of BigQuery)
  • storage.buckets.getIamPolicy
  • storage.buckets.get
  • storage.multipartUploads.abort
  • storage.multipartUploads.create
  • storage.multipartUploads.list
  • storage.multipartUploads.listParts
  • storage.objects.create
  • storage.objects.delete
  • storage.objects.get
  • storage.objects.list
  • storage.objects.update

Advanced Features script

Cloudability 's Service Account must have access to specific GCP APIs to provide Advanced Features support.

The script performs two steps: it first sets up a custom role within the project and then adds our Service Account as a member of the project, binding the custom role. This permission is scoped to a specific project and does not give Cloudability access to all projects.

1. Custom role setup

The script first creates a custom role within the project. Following the Principle of Least Privilege, the script attaches the minimum permissions necessary to support Advanced Features to the custom role. Currently, these permissions are:
  • compute.commitments.get
  • compute.commitments.list
  • compute.instances.list
  • monitoring.timeSeries.list
  • compute.disks.list
  • recommender.computeInstanceMachineTypeRecommendations.list
  • recommender.computeInstanceMachineTypeRecommendations.get
  • recommender.computeInstanceIdleResourceRecommendations.list
  • recommender.computeInstanceIdleResourceRecommendations.get

The list of permissions can change over time as we introduce new features that require additional permissions. Consequently, the script may change to improve UX through better error handling and clear messaging. 

# Example: Create advanced features custom role for my-project-123 # Project ID is my-project-123 gcloud iam roles create CloudabilityRole_AdvancedFeatures \ --project \ my-project-123 \ --title \ "Cloudability Advanced Features Role" \ --description \ "Allows Cloudability access to project level data for advanced features and analytics" \ --permissions \ compute.commitments.get,compute.commitments.list \ --stage=GA

2: Add Service Account as Member and Bind Custom role

Once the custom role has been created, the script adds Cloudability 's Service Account as a member of the project and binds the custom role to it. 

# Example: Add Cloudability's Service Account as member of my-project- 123 # Project ID is my-project-123 gcloud projects add-iam-policy-binding staging-vpc \ --member serviceAccount:billing-data-service-acct@cloudability -credentials.iam.gserviceaccount.com \ --role 'projects/my-project- 123/roles/CloudabilityRole_AdvancedFeatures'

Steps for integration

Create a storage bucket

A Storage bucket is required to copy data from the BQ table export into, for Cloudability to then retrieve this data.

Working in the Google Console ; type Cloud Storage in the Search bar and select it when it shows.

Click Create to create a Bucket.

Enter a name for your storage bucket (eg: cloudability-export and click Create).

When prompted keep defaults and click Confirm.

Bucket will be created.

�

Configure an account credential

This guide will help you through the process of credentialing your organization's billing account. This enables Cloudability to start ingesting your Google Cloud Platform (GCP) billing data.

Note:

If your organization has multiple GCP billing accounts, then this process must be repeated for each of those billing accounts.

The general process is as follows:

  1. Review prerequisites

  2. Add a new account credential

    Create a new GCP role-based billing account credential

  3. Run script

    Cloudability uses your billing Table ID to generate a shell script that uses IAM to:

    • Add a custom role to your billing project
    • Add Cloudability 's Service Account as a member of your billing project with that custom role
  4. Verify credentials

    Verify that our Service Account has access to read your billing data

Add a new account credential for Standard Billing
Follow the below steps to get started on creating a new GCP role-based credential for your billing account.
  1. In Cloudability , navigate to Settings > Vendor Credentials > Add Datasource > GCP. The Add GCP Account panel opens.

    Or

    In Cloudability , navigate to Settings > Vendor Credentials > GCP . Select Add a Credential . The Add a Credential panel opens.

  2. Select GCP Standard Billing from the drop down.
  3. Enter the billing Table ID .

  4. Enter the GCS Bucket (Optional).

    Note:

    GCS bucket which we created before is an optimized way to pull your data.

  5. Enter the GCP Organization Id (Optional).

    Note:

    GCP customers of Cloudability can quickly credential the projects in an automated way by using GCP organization . You will need to move any projects you created under "No organization" into your new organization resource.

    For instructions on how to move your projects, see Migrating projects into an organizational resource .

  6. Select Generate Setup Script , and then select Download Script .
  7. If your browser prompts you with a warning when downloading the shell script, select Keep .
Add a new account credential for Detailed Billing
Follow the below steps to get started on creating a new GCP role-based credential for your billing account:
  1. In Cloudability , navigate to Settings > Vendor Credentials > Add Datasource > GCP . The Add GCP Account panel opens.

    Or

    In Cloudability , navigate to Settings > Vendor Credentials > GCP . Select Add a Credential . The Add a Credential panel opens.

  2. Select GCP Detailed Billing from the drop down.
  3. Enter the billing Table ID .
  4. Enter the GCS Bucket .

    Note:

    Ensure that you have configured the GCS permissions as mentioned above.

  5. Enter the Detailed Billing Date .
    Note:

    This date indicates the month and the year from when you have detailed billing exports enabled. If you are doing this for the first time it is the current month as YYYY-MM.

  6. Enter the GCP Organization Id (Optional).

    Note:

    GCP customers of Cloudability can quickly credentialize the projects in an automated way by using GCP organization . You will need to move any projects you created under "No organization" into your new organization resource.

    For instructions on how to move your projects, see Migrating projects into an organizational resource .

  7. Select Generate Setup Script , and then select Download Script .
  8. If your browser prompts you with a warning when downloading the shell script, select Keep .

    Note:

    Same GCP Payer Account IDs cannot be used in both standard and detailed billing at the same instance.

Run the script

Follow the below steps to run the script via the Cloud Shell within your cloud console. It does not matter from where within the Cloud Shell you run the script.
  1. Activate the Cloud Shell, select the icon and select Upload file . Choose the downloaded script from the file explorer and confirm.
  2. Run the script from within the Cloud Shell.

    Note:

    In the commands below, replace script-name with the full name and extension of the actual script.

    You must let the Cloud Shell know that the script is executable. You can do this by running the following command in your Cloud Shell:

    chmod +x script-name

    Next, run the script in your Cloud Shell:

    ./ script-name
  3. Check for any errors. For example:

    If the script is successful, you will see a similar output to the following in your Cloud Shell:

Verify credentials
  1. Select Verify Credentials .

    Your billing account is added and a green notification indicates that the credential was successfully verified.

  2. Select the refresh icon to update the status.

    The green check-mark indicates that this billing account has been successfully credentialed.

You have now successfully added your billing account to Cloudability. We ingest data at regular intervals and your billing data will be available starting from the next ingest cycle. Upon the next ingest, we will also enumerate the projects associated with this billing account. Return to the GCP credentials page to see the list of projects linked to the billing account.

Lastly, if your organization has additional GCP billing accounts that you would like to add, then please repeat this process for each of those billing accounts.

Note: While setting up your GCP accounts, turn off any applied domain restrictions. You can turn it on once the credentialing process is completed.

Configure project-level credentials

The purpose of this section is to help you walk through the process of credentialing your projects to enable Cloudability's Advanced Features. If your organization has multiple GCP projects, then this process must be repeated for each project for which you would like to enable Advanced Features.

Before you begin, make sure you’ve completed the prerequisites that are listed here .

To credential your project:

  1. Add a new project credential, as described in Configure project-level credentials.

  2. Run the script, as described in Run the script .

  3. Verify the credentials, as described in Verify credentials .

Add a new project credential

You can use the following steps to enable Advanced Features for a project. You will be required to run a script within your Google cloud shell. This script provisions Cloudability's Service Account with the permissions necessary to support Advanced Features.

Note:

In the project details pane, the disabled Advanced Features are marked with x.

To enable Advanced Features for a GCP project:
  1. Navigate to Vendor Credentials , and select the GCP tab

  2. Select the Edit icon to open Edit a Credential panel.

  3. Enter Organization ID .(Optional)
    Note:

    GCP customers of Cloudability can quickly credentialize the projects in an automated way by using GCP organization . You will need to move any projects you created under "No organization" into your new organization resource.

    For instructions on how to move your projects, see Migrating projects into an organizational resource .

  4. Select Download Script .
  5. If your browser prompts you with a warning, select Keep .

Run the script

For more information, see Run the script .

How to confirm success

Select Verify Credentials .

Select Details to verify that the project has the necessary permissions for Advanced Features.

Enhancement for Cloudability Premium customers

Upon upgrading to Cloudability Premium , the Billing Reports and Advanced Reports status for each GCP account in the listing page automatically change to error status as the GCP token mechanism is changed with this release. Hence, Cloudability admin needs to edit each account following the steps below which will set the right account status in Cloudability for GCP data ingestion as well as enable Cloudability share these accounts with Turbonomic .

  1. Click Optimize Resources in the toggle button.

  2. Generate setup script.

  3. Update the permissions by executing the script.

  4. Re-verify the account.

There are additional Turbonomic permissions that gets added to basic (Billing Data), advanced (Utilization Data) and Optimize Resources (execute actions) which are documented in the help center documents. Once your account is verified, the list of permissions can be viewed by choosing the Details option on each GCP account listed under Cloudability.

Troubleshooting

Error: The details provided on the Billing Export page differ from those on the BigQuery page for your Billing table.

Specifically, the Table ID on the BigQuery page is constructed using the Billing Project ID, while the Billing Export page lists the Billing Project Name. IDs are unique, while names are not.

Solution: Review prerequisites here. For more details on GCP Projects, see REST Resource: projects.

[UI] Truncation: Project ID, Dataset, or Table ID getting truncated.

Context: You enter your complete GCP Billing Table ID into and notice that there are errors. Additionally, when you edit the credential, you notice that the Project ID, Dataset, or Table ID are truncated. You might have retrieved or constructed your GCP Billing Table ID from the Billing Export page.

Solution: You must grab the complete Table ID from the BigQuery page for your Billing table.

[Script] Error: (gcloud.iam.roles.create) A resource in the project is a subject of conflict.

Context: This can occur when you have an existing role within your billing project with role_id CloudabilityRole_Billing. The error indicates that the script is unable to create a new role with role_id CloudabilityRole_Billing because one already exists.

Solution: Ignore this error.

[Script] Error: (gcloud.iam.roles.create) FAILED_PRECONDITION: You can't create a role with role_id (CloudabilityRole_Billing) where there is an existing role with that role_id in a deleted state.

Context: This can occur when you run the script after deleting an existing role from your billing project with role_id CloudabilityRole_Billing. The role could be in a deleted state and the script can't create a new role with that role_id. You can view the role's status (Enabled, Disabled, Deleted) in your cloud console.

Solution: Undelete the existing role with role_id CloudabilityRole_Billing from your billing project and re-run the script.

Getting Recommendations Based on GPU Data

GCP Domain Restrictions

If you have domain restrictions, then you might observe error messages. To overcome this, whitelist Cloudability to query the data by adding Cloudability 's GCP workspace customer id in their Organisation policy.

Reach out to support team if you need Cloudabilitiy's GCP workspace customer id.

VPC Service Controls

Getting Recommendations Based on GPU Data

Prerequisites:
  1. Each VM must have GPUs attached .
  2. Each VM must have a GPU driver installed .
  3. Each VM must have Python 3.6 or newer installed.
  4. Each VM must have the packages required for creation of Python virtual environments installed.

Also, see Set up GCP Monitoring Agent for rightsizing .