Cloudability Connect Databricks

Edit online

You can connect your Databricks account to Cloudability to enable the ingestion of cost and usage data. This integration is intended to be used only for Databricks on AWS and Databricks on GCP. If you are using Azure Databricks, you will have sufficient data granularity in your Azure billing data.

It takes 4 to 24 hours before your initial cost and usage data appears in Cloudability. During this time, a message indicates that the process is yet to complete.

Summary of the integration

This integration requires users to set up a Unity Catalog, enable a Workspace & System Schema and create a Service Principal User that will have access to the catalog, schema, and table. Users also need to create a SQL Warehouse, that can be used by the Service Principal User to run it, based on the notebook downloaded in the Credentials UI. This notebook must run once every day to pull Databricks costs and usage data. Cloudability does not have access to your custom discounts with Databricks, so we have added an option to input that information in the credentialing UI so that we can factor this discount into the costs we display.

Databricks recommends using a service principal identity for automated tools, jobs, and applications. We ask you to create one for this integration, as we are automating the process of usage and billing monitoring.

Note:

It If you are purchasing Databricks via a cloud provider marketplace and adding cost and usage data for Snowflake with this integration, you will see costs displayed twice in Cloudability reporting. As an example, if you purchased Snowflake via the AWS Marketplace, you would have:

  • The high-level line item for AWS.

  • The line items with detailed Snowflake costs and AWS Marketplace listed as Seller.

You will need to set up filters or views to hide your Marketplace costs. Marketplace costs are excluded from billing.

Before you start

  • You are a Cloudability administrator.

  • You are a Databricks Account admin to manage OAuth credentials for the service principals.

Steps for integration

Integration Pre-requisites

  1. Set Up Unity Catalog. Click https://docs.databricks.com/en/data-governance/unity-catalog/get-started.html .

  2. Enable a Workspace for Unity Catalog. Click https://docs.databricks.com/en/data-governance/unity-catalog/enable-workspaces.html .

  3. Enable System Schema in Unity Catalog.

    Click https://docs.databricks.com/api/azure/workspace/systemschemas/enable

  4. Create a Service Principal User at account level. Click https://docs.databricks.com/en/admin/users-groups/service-principals.html#add-service-principals-to-your-account-using-the-account-consol .

Save the Client Id and Secret of the Service Principal.

  1. Add Service Principal to Unity Catalog Enabled Workspace. Click https://docs.databricks.com/en/admin/users-groups/service-principals.html#add-a-service-principal-to-a-workspace-using-the-workspace-admin-settings .

  2. Download Notebook from credentials page and run it and schedule a job to run it once everyday.

  3. Grant Service Principal the privilege on the newly created catalog, schema and table. Click https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html#manage-privileges-in-unity-catalog .

  4. Create an SQL warehouse, Click https://docs.databricks.com/en/compute/sql-warehouse/create.html .

SQL Warehouse should be serverless.

  1. Grant Service Principal the permission to use SQL Warehouse. Click https://docs.databricks.com/en/compute/sql-warehouse/create.html#manage-a-sql-warehouse .

  2. The downloaded notebook must run once every day.

Create a Service Principal

To add a service principal to the account using the account console, follow the steps below.
  1. As an account admin, log in to the account console.
  2. Click User Management .
  3. On the Service Principals tab, click Add Service Principal .
  4. Enter a name for the service principal.
  5. Click Add .
Create permissions for service principal
  1. As an account admin, log in to the account console.
  2. Click User Management .
  3. On the S ervice Principals tab, select your service principal.
  4. Click the Roles tab.
  5. Opt in for Account Admin role
Create an OAuth secret for a service principal
  1. As an account admin, log in to the account console.
  2. Click User Management .
  3. On the Service Principals tab, select your service principal.
  4. Under OAuth Secrets , click Generate Secret .
  5. Copy the displayed Secret and Client ID , and then click Done.
Add Service Principal to the Workspace
  1. Log in to the workspace.
  2. Click User Management .
  3. Select Settings .
  4. Click Identity and access .
  5. Click Manage in Service principals.
  6. Click Add Service Principal.
  7. Add the service principal.
Add Permission to Service Principal to the Warehouse
  1. Click SQL Warehouse .
  2. For the warehouse to be used in querying, click on the warehouse.
  3. Click Permissions .
  4. Add Service Principal with the Can Use permission.
Add Permission to Service Principal to the Table
  1. Click on the catalog.
  2. Click the catalog name, for which permissions need to be given.
  3. Click Permissions .
  4. Click Grant .
  5. Select the Principals.
  6. Select USE CATALOG , USE SCHEMA , and SELECT under Privileges .
  7. Click Grant .
Note:

Private links are not yet supported for Databricks.