Working with watsonx.data APIs

watsonx.data APIs in the developer edition can be accessed using a bearer token for authentication. This topic outlines the steps to generate the token and use it to invoke APIs, such as the Ingestion API.

Generate Bearer Token

Run the following curl command to generate bearer token:

 curl --location -k 'https://localhost:6443/lakehouse/api/v3/auth/authenticate' \
--header 'Content-Type: application/json' \
--data '{
    "username": "ibmlhadmin",
    "password": "password",
    "instance_id": "0000-0000-0000-0000",
    "instance_name": ""
}'

Invoking the Ingestion API

Run the following command to invoke the Ingestion API.

Sample request:

curl --location -k 'https://localhost:6443/lakehouse/api/v3/lhingestion/api/v1/ingestions' \
--header 'Accept: application/json' \      
--header 'Authinstanceid: 0000-0000-0000-0000' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $token' \ 
--data '{               
    "target": {
        "catalog": "iceberg_data",
        "schema": "schema1",                                                  
        "table": "test1"   
    },                     
    "source": {                          
        "file_paths": "s3://hive-bucket/wxd-local-ingestion/2/jsonFile.json", 
        "file_type": "json",
        "bucket_details": {
            "bucket_name": "hive-bucket",
            "bucket_type": "minio"
        }              
    },                        
    "job_id": "ingestion-test-01",
    "engine_id": "spark158", 
    "execute_config": {     
        "driver_memory": "0G",
        "driver_cores": 0,
        "executor_memory": "4G",
        "executor_cores": 2,
        "num_executors": 1
    }
}'

Sample response:

{"instance_id":"0000-0000-0000-0000","job_id":"ingestion-test-01","application_id":"8418c042-645b-4125-94dc-fe9871f2b203","username":"ibmlhadmin","start_timestamp":"1759247749366117610","status":"starting","source_data_files":"s3://hive-bucket/wxd-local-ingestion/2/jsonFile.json","target_table":"iceberg_data.schema1.test1","details":"Ingestion Parameters:\n{\n  
\"job\" : {\n    \"id\" : \"ingestion-test-01\",\n    \"crn\" : \"0000-0000-0000-0000\",\n    \"username\" : \"ibmlhadmin\",\n    \"status\" : null,\n    \"startTimestamp\" : \"1759247749366117610\",\n    \"endTimestamp\" : null\n  },\n  \"engine\" : {\n    \"endpoint\" : \"https://lhconsole-ui-svc:443/lakehouse/api/v3/spark_engines/spark158/applications\",\n    \"origin\" : \"native\",\n    \"execute_config\" : {\n      
\"driver_memory\" : \"0G\",\n      \"driver_cores\" : 0,\n      \"executor_memory\" : \"4G\",\n      \"executor_cores\" : 2,\n      \"num_executors\" : 1\n    },\n    \"bucket_details\" : {\n      \"endpoint\" : \"http://ibm-lh-minio-svc:9000\",\n      \"region\" : \"us-east-1\",\n      \"auth_mode\" : \"hmac\",\n      \"modified_at\" : \"1759223796\",\n      \"name\" : \"iceberg-bucket\",\n      \"type\" : \"minio\"\n    },\n    \"log_path\" : \"s3://iceberg-bucket/tmp_watsonx_data_ingestion_spark/logs/ingestion-test-01.log\"\n  },\n  \"source\" : {\n    \"file_paths\" : \"s3://hive-bucket/wxd-local-ingestion/2/jsonFile.json\",\n    \"file_type\" : \"json\",\n    \"source_type\" : \"storage\",\n    \"bucket_details\" : {\n      \"endpoint\" : \"http://ibm-lh-minio-svc:9000\",\n      \"region\" : \"us-east-1\",\n      \"auth_mode\" : \"hmac\",\n      \"managed_by\" : \"ibm\",\n      \"modified_at\" : \"1759223796\",\n      \"associated_catalogs\" : [ {\n        \"name\" : \"hive_data\",\n        \"base_path\" : \"/\"\n      } ],\n      \"name\" : \"hive-bucket\",\n      \"type\" : \"minio\"\n    },\n    \"local_ingestion\" : false,\n    \"is_local_ingestion\" : false\n  },\n  \"target\" : {\n    \"catalog\" : \"iceberg_data\",\n    \"schema\" : \"schema1\",\n    \"table\" : \"test1\",\n    \"write_mode\" : \"append\",\n    \"schema_mode\" : \"strict\",\n    \"catalog_uri\" : \"https://ibm-lh-mds-thrift-svc:8381/mds/thrift?catalog=iceberg_data\",\n    \"bucket_details\" : {\n      \"endpoint\" : \"http://ibm-lh-minio-svc:9000\",\n      \"region\" : \"us-east-1\",\n      \"auth_mode\" : \"hmac\",\n      \"modified_at\" : \"1759223796\",\n      \"name\" : \"iceberg-bucket\",\n      \"type\" : \"minio\"\n    },\n    \"location\" : \"s3a://iceberg-bucket/schema1\",\n    \"new_schema\" : false,\n    \"new_table\" : false,\n    \"is_new_schema\" : false,\n    \"is_new_table\" : false\n  },\n  \"job_id\" : 
\"ingestion-test-01\"\n}\n","engine_logs":"\n","new_schema":false,"new_table":false,"is_new_schema":false,"is_new_table":false}