Azure Data Lake Storage
Azure Data Lake Storage (ADLS) is a scalable data storage and analytics service that is hosted in Azure, Microsoft's public cloud. The Microsoft Azure Data Lake Storage connection supports access to both Gen1 and Gen2 Azure Data Lake Storage repositories.
watsonx.data on Red Hat® OpenShift®
watsonx.data Developer edition
Select Azure Data Lake Storage from the Storage section of the Add component window.
If you select Azure Data Lake Storage Gen1 Blob from the Type drop-down, configure the following details:
| Field | Description |
|---|---|
| Display name | Enter the name to be displayed. |
| Container name | Enter the Container name. |
| Storage account name | Enter the Storage account name. |
| Endpoint | The Endpoint URL is auto-generated. |
| Access key | Enter the access key. |
| Associate catalog | Select the checkbox to add a catalog for your storage. This catalog is associated with your storage and serves as your query interface with the data stored within. |
| Catalog type | Select the catalog type from the list. The recommended catalog is Apache Iceberg. The other options for catalog are Apache Hive, Apache Hudi, and Delta Lake. |
| Catalog name | Enter the name of the associated catalog. |
| Create | Click Create to create the storage. |
If you select Azure Data Lake Storage Gen2 from the Type drop-down, configure the following details:
| Field | Description |
|---|---|
| Display name | Enter the name to be displayed. |
| Container name | Enter the Container name. |
| Storage account name | Enter the Storage account name. |
| Endpoint | The Endpoint URL is auto-generated. |
| Application id | Enter the Application id. |
| Directory id | Enter the Directory id. |
| Secret key | Enter the Secret key. |
| Associate catalog | Select the checkbox to add a catalog for your storage. This catalog is associated with your storage and serves as your query interface with the data stored within. |
| Catalog type | Select the catalog type from the list. The recommended catalog is Apache Iceberg. The other options for catalog are Apache Hive, Apache Hudi, and Delta Lake. |
| Catalog name | Enter the name of the associated catalog. |
| Create | Click Create to create the storage. |
Limitations
- Use separate containers and storage accounts for ADLS Gen1 and ADLS Gen2 storage for complete metadata synchronization, including tables. Otherwise, a PARTIAL SUCCESS message appears in the sync logs when SYNC finishes.