Querying Databricks Delta Share tables using Spark engine

Create a PySpark script using the following template and upload it to your storage bucket. The script reads a Delta Sharing table using the deltaSharing format and the path constructed from your profile file and table coordinates.

"""
Delta Share reader for IBM watsonx.data Spark engine.
Reads shared Delta tables using the Delta Sharing open protocol.
"""

from pyspark.sql import SparkSession


def read_delta_share_table(spark, config_path, share_name, schema_name, table_name):
    """
    Read a Delta Share table.

    Args:
        spark: SparkSession provided by the watsonx.data Spark engine
        config_path: Full path to the .share profile file
                     CP4D example:  /mnts/conf/config.share
                     SaaS example:  s3a://<bucket>/config.share
        share_name:  Name of the Delta Share
        schema_name: Schema name within the share
        table_name:  Table name within the schema
    Returns:
        DataFrame containing the shared table data
    """
    # Table path format: <config_path>#<share_name>.<schema_name>.<table_name>
    table_path = f"{config_path}#{share_name}.{schema_name}.{table_name}"

    print(f"Reading Delta Share table: {table_path}")
    df = spark.read.format("deltaSharing").load(table_path)

    print(f"Rows: {df.count():,}  Columns: {len(df.columns)}")
    df.printSchema()
    return df


def main():
    CONFIG_PATH  = "<path_to_profile_file>/config.share"
    SHARE_NAME   = "<share_name>"
    SCHEMA_NAME  = "<schema_name>"
    TABLE_NAME   = "<table_name>"

    spark = SparkSession.builder.getOrCreate()
    print(f"Spark version: {spark.version}")

    df = read_delta_share_table(
        spark=spark,
        config_path=CONFIG_PATH,
        share_name=SHARE_NAME,
        schema_name=SCHEMA_NAME,
        table_name=TABLE_NAME,
    )

    print("Sample data (first 10 rows):")
    df.show(10, truncate=False)
    return df


if __name__ == "__main__":
    df = main()

Replace the following placeholders in the script:

• <path_to_profile_file>: Mount path of the storage volume containing the profile file (CP4D), or the S3 path (SaaS). Example CP4D value: /mnts/conf.
• <share_name>: Name of the Delta Share as configured in Databricks.
• <schema_name>: Schema name within the share.
• <table_name>: Table name within the schema.