Azure Data Lake Storage

Azure Data Lake Storage (ADLS) is a scalable data storage and analytics service that is hosted in Azure, Microsoft's public cloud. The Microsoft Azure Data Lake Storage connection supports access to both Gen1 and Gen2 Azure Data Lake Storage repositories.

Important: Azure Data Lake Storage (ADLS) Gen1 is deprecated and will be removed in an upcoming release. You must transition to ADLS Gen2 as ADLS Gen1 will no longer be available.

watsonx.data on IBM Software Hub

watsonx.data Developer edition

Select Azure Data Lake Storage from the Storage section of the Add component window.

If you select Azure Data Lake Storage Gen1 Blob from the Type drop-down, configure the following details:

Field	Description
Display name	Enter the name to be displayed.
Container name	Enter the Container name.
Storage account name	Enter the Storage account name.
Endpoint	The Endpoint URL is auto-generated.
Input method Note: This field is available only when use of secrets from an external vault (HashiCorp) is configured for your connections.	Choose the credential input method: Enter credential manually Use secret from vault To use secrets from an external vault, select the Use secret from vault option and complete the following steps: Click Select secret. Select one or more secrets and click Select.
Access key	If you are using secrets from vault, then select the Access key from the drop-down list. Otherwise, enter your Access key.
Connection status	Click the Test connection link to test the bucket connection. If the bucket connection is successful, a success message appears.
Console UI Associate catalog	Console UI Select the checkbox to associate a catalog to the data source. This catalog is automatically associated with your data source and serves as your query interface with the data stored within.
Catalog type / Type	Console UI Catalog type: Select the catalog type from the list. The recommended catalog is Apache Iceberg. The other option for catalog is Apache Hive. Platform UI Type: Select the database type from the list. The recommended database is Apache Iceberg. The other option for database is Apache Hive.
Catalog name / Name	Console UI Catalog name: Enter the name of the catalog. Platform UI Name: Enter the name of the database associated with your data source. The database serves as your query interface with the data stored within.
Base path (optional) Note: This field appears only when you select Apache Iceberg as the data source type.	Enter the base path for the catalog in the object storage. This allows you to associate multiple Iceberg data sources with a single storage. Note: You cannot share a storage between Iceberg and non-Iceberg data sources.
Associate	Click Associate to create the storage.

If you select Azure Data Lake Storage Gen2 from the Type drop-down, configure the following details:

Field	Description
Display name	Enter the name to be displayed.
Container name	Enter the Container name.
Storage account name	Enter the Storage account name.
Endpoint	The Endpoint URL is auto-generated.
Authentication Mode	Based on your requirement, select one of the following modes of authentication: Account Key: If you select Account Key, enter the access key in the Access Key field. Service Principal: If you select Service Principal, enter the Application id, Directory id, and Secret key.
Connection status	Click the Test connection link to test the bucket connection. If the bucket connection is successful, a success message appears.
Console UI Associate catalog	Console UI Select the checkbox to associate a catalog to the data source. This catalog is automatically associated with your data source and serves as your query interface with the data stored within.
Catalog type / Type	Console UI Catalog type: Select the catalog type from the list. The recommended catalog is Apache Iceberg. The other option for catalog is Apache Hive. Platform UI Type: Select the database type from the list. The recommended database is Apache Iceberg. The other option for database is Apache Hive.
Catalog name / Name	Console UI Catalog name: Enter the name of the catalog. Platform UI Name: Enter the name of the database associated with your data source. The database serves as your query interface with the data stored within.
Base path (optional) Note: This field appears only when you select Apache Iceberg as the data source.	Enter the base path for the data source in the object storage. This allows you to associate multiple Iceberg data sources with a single storage location. Note: You cannot share a storage location between Iceberg and non-Iceberg data sources.
Associate	Click Associate to create the storage.

Limitations

Use separate containers and storage accounts for ADLS Gen1 and ADLS Gen2 storage for complete metadata synchronization, including tables. Otherwise, a PARTIAL SUCCESS message appears in the sync logs when SYNC finishes.