Working with delivery methods

Data Product Hub provides several options for delivering data products to the consumer. Each item in a data product can be delivered by using different methods. The connector type determines the available delivery methods.

Cloud platforms

Configuring delivery methods

To enable or disable a delivery method, go to Configuration and settings > Delivery and click the toggle switch for the delivery method.

Types of delivery methods for Data Product Hub

You can configure your data product to be delivered by the following delivery methods, depending upon the connector type:

  • Custom delivery method
  • Downloading directly from a URL
  • Downloading a data extract from a target connected data source
  • Opening a URL
  • Accessing a data product with Flight service
  • Access in watsonx.data
  • Deliver to watsonx.data
  • Access in Azure Databricks
  • Access in Snowflake
  • Access in AWS Databricks
Types of delivery methods
Name Description Notes
Custom delivery method Determines how consumers receive their data product. Upon subscription, the consumer will see the information required to access the data in the subscription panel.
Download Consumers receive a URL to download a data product from a connection. Requires a data source connection that supports download.
Data extract Consumers can download a data product as a file extract from a target connection. Requires two connections: A source connection and a target connection. Also requires a default project.
Open URL Consumers receive a URL to directly access a data product. Does not require a data source connection.
Live access with Flight service Consumers receive a code snippet to connect with a data source and receive a data product. The provided code snippet can be added to notebooks and other applications.
Access in watsonx.data Consumers can access the data product that is created from the watsonx.data instance. Requires access to watsonx.data and a Presto engine
Deliver to watsonx.data Consumers access their data product as a table in watsonx.data. Does not support FIPS enabled cluster in 5.2.0
Access in Azure Databricks Consumers access their data product as a table in Azure Databricks. Consumers must have access to Azure Databricks.
Access in Snowflake Consumers receive their data product as a table that was created in Snowflake.
Access in AWS Databricks Consumers receive their data product as a table that was created in AWS Databricks. Consumers must have access to AWS Databricks.

The data extract, live access with flight service, and deliver to watsonx.data delivery methods support the following source data formats: CSV, Excel, Avro, Delimited, JSON, Parquet, ORV, SAV, XML and Tables in relational data sources.

Download

The Download delivery method generates a URL that allows consumers to download directly from a connected data source.

Open URL

The Open URL delivery method provides a URL that points to a website containing the items in a data product.

Data extract

You can extract data from a database and deliver the extract as a file to consumers. The data extract delivery method requires two connections, one for connecting to the database you are extracting from (the source) and another connection for delivering the file to the consumer (the target). To view which connectors support source and target connections for data extracts, see Delivery methods for connectors.

The data extract delivery method requires a default project. The default project is created automatically for you during installation.

The target connection is a file storage location that is used exclusively for storing data extracts. This connection delivers the extract to consumers through a download URL. Since data extracts can be large, a file storage location with adequate capacity is recommended. The target connections require read/write credentials.

Create two connections for the data extract delivery method:

  • Source connection: The source connection is the location where you extract data.
  • Target connection: The target connection is the location where the extracted file is stored and where consumers download a data extract from a URL.

The data extract delivery method supports the following target file types: Avro, CSV, Delimited text, Excel, JSON, ORC, Parquet, SAV, and XML.

Live access with Flight service

The Flight service provides a single interface for real-time access to many different data sources through a common open source API. Access to the data source through the Flight service delivery method is governed by the delivery credentials that are specified by the Producer when the data product was created. The delivery credentials typically have read-only access to the data source to ensure that consumers can only read the data on subscribing. To determine whether your data source supports the Flight service, see Delivery methods for connectors.

When you subscribe to a data product and select delivery by the Flight service, a Flight URL is provided. You download a code snippet in your preferred language (Python or R). The code snippet contains an asset ID and a catalog ID to connect to a data source to receive delivery of the items in a data product. The code snippet can be inserted into a notebook and other applications.

You can also access the data programmatically by using an Arrow client. Arrow libraries are available for C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, and Ruby. See Apache Arrow for instructions on installing the libraries for each language.

See the Flight client example for an example of how to access your data product by using Python.

By default, the Flight service is only available to the Cloud Pak for Data instance that is running it. If you want to deliver data products by using the Flight service from an external data source, the Cloud Pak for Data administrator must configure an external route. For details, see Configuring an external route to the Flight service in the IBM Software Hub documentation.

For more information about Arrow Flight RPC, see Arrow Flight RPC.

Access in watsonx.data

The access in watsonx.data delivery method provides consumers access to a data product containing data assets from a watsonx.data lakehouse.

Pre-requisites for Data Producers:

A data producer must have the following roles to create a data product from data assets in watsonx.data.

Required roles:
Data Product Hub community role: Editor or Administrator
watsonx.data lakehouse roles:
watsonx.data instance role: User
Catalog role: Admin
Presto engine role: User

A data producer can enable this delivery method by using the following steps:

  1. Create a watsonx.data Presto connection in Data Product Hub. The credentials in the Presto connection will be used to create an access policy in watsonx.data to grant consumers access to a specific table.
  2. Create a data product by including data assets by using the watsonx.data Presto connection.
  3. Select the Access in watsonx.data delivery method. During subscription, the consumer is granted access to the table in watsonx.data lakehouse.
  4. Optional: Allow consumers access to a Presto engine.
  5. Publish the data product.

Prerequisites for Data Consumers

A consumer must have the following roles to subscribe to a data product and access the data in watsonx.data.

Required roles:
Data Product Hub community roles: Editor or Viewer
watsonx.data lakehouse role: User

Consumers can subscribe to a data product with this delivery method by using the following steps:

  • watsonx.data console link: For easy access to the delivered data in watsonx.data.
  • Data asset: Specific catalog, schema, and table details, which can be viewed in the Data manager view in watsonx.data
  • Engine Selection: You can choose Presto, Spark, or both. If the data producer enabled Presto or Spark during data product creation, you can use the selected engine to run SQL queries on the table.

After you subscribe to the data product, setup instructions will be displayed and you can copy the link of the following preferred access options:

  • SQL
  • REST
  • Compute Engine
  • JDBC/ODBC

More details of the of the subscription can be obtained in the My Subscriptions page.

Deliver to watsonx.data

The deliver to watsonx.data delivery method allows consumers to access their data product as a table within their watsonx.data instance. It works by storing the data as a CSV file in an IBM Cloud Object Storage bucket, and then by using an ingest job in watsonx.data to load the CSV into the specified table. Users with the appropriate permissions can create new tables or append data to existing ones. The deliver to watsonx.data delivery method requires a default project.

Pre-requisites for Data Consumers

Required roles
Data Product Hub community role: Editor
watsonx.data lakehouse roles:

Service access roles: MetastoreAdmin and DataAccess

Platform access role: Administrator

Target iceberg catalog: User
You can also create an access policy that gives some permission to use the catalog. For more information, see Managing data policy in the watsonx.data documentation.

Spark engine connected to the target catalog: User

Presto engine role: User

Storage used for the target catalog: Writer or above

Cloud Object Storage: Add as a component in the watsonx.data lakehouse. The data extract will land here before being ingested to watsonx.data.

Consumers must create two connections in Data Product Hub:

  1. Cloud Object Storage: Add a connection by using the same connection as watsonx.data. You must use HMAC credentials to configure the Access key and Secret key or the Service credentials authentication method.
  2. IBM watsonx.data Presto: Copy the JSON snippet provided in the watsonx.data Presto engine instance and pasting it when you are creating a new connection.

Access in Azure Databricks

The Access in Azure Databricks delivery method provides consumers access to a data product containing data assets from Azure Databricks.

A cluster administrator must configure egress to allow external API calls to Azure Databricks. For more details, see Define custom egress network policy.

Prerequisites for Data Producers:

A data producer must have the following roles to create a data product from data assets in Azure Databricks.

Required roles:
Data Product Hub community role: Editor or Administrator
Azure Databricks roles:
Access to the SQL Warehouse in Databricks
ALL PRIVILEGES or MANAGE access on the catalog, schema, table

A data producer can enable this delivery method by using the following steps:

  1. Create an Azure Databricks connection in Data Product Hub. The credentials in the connection grants consumers access to a specific table in Azure Databricks.
  2. Create a data product by including data assets by using the connection created in step 1.
  3. Select the Access in Azure Databricks delivery method. During subscription, the consumer is granted access to the table in Azure Databricks.
  4. Publish the data product.

Prerequisites for Data Consumers:

A consumer must have the following roles to subscribe to a data product and access the data in Azure Databricks

Required roles:
Data Product Hub community roles: Editor or Viewer
Azure Databricks: Access to the SQL Warehouse

Consumers can subscribe to a data product with this delivery method and access the data asset by using the following step:

  1. Specific catalog, schema, and table can be viewed in the Databricks UI by selecting Catalog in the left window and navigating to the schema and table.
  2. The details can be obtained from the subscription in the My Subscriptions page.

Access in AWS Databricks

The Access in AWS Databricks delivery method provides consumers access to a data product containing data assets from AWS Databricks.

A cluster administrator must configure egress to allow external API calls to AWS Databricks. For more details, see Define custom egress network policy.

Prerequisites for Data Producers:

A data producer must have the following roles to create a data product from data assets in AWS Databricks:
: Data Product Hub community role: Editor or Administrator : AWS Databricks roles: : Access to the SQL Warehouse in Databricks : ALL PRIVILEGES or MANAGE access on the catalog, schema, table

A data producer can enable this delivery method by using the following steps:

  1. Create an AWS Databricks connection in Data Product Hub. The credentials in the connection grants consumers access to a specific table in AWS Databricks.
  2. Create a data product by including data assets by using the connection created in step 1.
  3. Select the Access in AWS Databricks delivery method. During subscription, the consumer is granted access to the table in AWS Databricks.
  4. Publish the data product.

Prerequisites for Data Consumers:

A consumer must have the following roles to subscribe to a data product and access the data in AWS Databricks:
Data Product Hub community roles: Editor or Viewer
AWS Databricks: Access to the SQL Warehouse

After you subscribe to a data product:

  • Setup instructions are displayed and you can copy the link of the following preferred access options:
    • SQL
    • REST
    • Compute Engine
    • JDBC/ODBC
  • Add data product to a project

Consumers can subscribe to a data product with this delivery method and access the data asset by using the following step:

  1. Specific catalog, schema, and table can be viewed in the Databricks UI by selecting Catalog in the left window and navigating to the schema and table.
  2. The details can be obtained from the subscription in the My Subscriptions page.

Access in Snowflake

The Access in Snowflake delivery method provides consumers access to a data product containing data assets from Snowflake.

Prerequisites for data producers

A data producer must have the following roles to create a data product from data assets in Snowflake:
Data Product Hub community role: Editor or Administrator
Snowflake roles:
USAGE ON DATABASE WITH GRANT OPTION
USAGE ON SCHEMA WITH GRANT OPTION
SELECT ON TABLE WITH GRANT OPTION

A data producer can enable this delivery method by completing the following steps:

  1. From Data Product Hub, create a Snowflake connection. The credentials in the connection grant consumers access to a specific table in Snowflake.
  2. Create a data product by including data assets by using the connection created in step 1.
  3. Select the Access in Snowflake delivery method. During subscription, the consumer is granted access to the table in Snowflake.
  4. Publish the data product.

Important: When you configure the Access in Snowflake delivery method, Snowflake does not accept username and password as a valid authentication method for connecting to the REST API.

Prerequisites for data consumers

A consumer must have the following roles to subscribe to a data product that has assets in Snowflake:
Data Product Hub community role: Editor or Viewer

When you subscribe to a data product with the Access in Snowflake delivery method, you can choose the following authentication methods for each table provided:

  • Key-Pair
  • OAuth Client Credentials
  • Username and password

After you subscribe to a data product:

  • Consumers are granted secure and governed access to Snowflake.
  • Setup instructions are displayed and you can copy the link of the following preferred access options:
    • Snowflake UI Console URL
    • JDBC connection string
  • Add data product to a project

More details of the subscription can be obtained in the My Subscriptions page.

Learn more

Delivery methods for connectors