wx-data ingestion create

Create an ingestion job.

Syntax

cpd-cli wx-data ingestion create \
--engine-id=<engine-id> \
--target-table=<target-table> \   
[--bucket-access-key=<placeholder-text>] \
[--bucket-details=<placeholder-text>] \
[--bucket-endpoint=<placeholder-text>] \
[--bucket-name=<placeholder-text>] \
[--bucket-region=<placeholder-text>] \
[--bucket-secret-key=<placeholder-text>] \
[--bucket-type=<placeholder-text>] \
[--csv-encoding=<placeholder-text>] \
[--csv-escape-character=<placeholder-text>] \
[--csv-field-delimiter=<placeholder-text>] \
[--csv-header=<placeholder-text>] \
[--csv-line-delimiter=<placeholder-text>] \
[--csv-property=<placeholder-text>] \
[--database-auth-value=<placeholder-text>] \
[--database-cert-extension=<placeholder-text>] \
[--database-certificate=<placeholder-text>] \
[--database-connection-mode=<placeholder-text>] \
[--database-connection-mode=<placeholder-text>] \
[--database-host=<placeholder-text>] \
[--database-id=<placeholder-text>] \
[--database-isssl=<placeholder-text>] \
[--database-name=<placeholder-text>] \
[--database-password=<placeholder-text>] \
[--database-port=<placeholder-text>] \
[--database-schema=<placeholder-text>] \
[--database-table=<placeholder-text>] \
[--database-type=<placeholder-text>] \
[--database-user-id=<placeholder-text>] \ 
[--database-warehouse=<placeholder-text>] \
[--driver-cores=<placeholder-text>] \
[--driver-memory=<placeholder-text>] \ 
[--execute-config=<placeholder-text>] \
[--executor-cores=<placeholder-text>] \
[--executor-memory=<placeholder-text>] \
[--iceberg-catalog=<placeholder-text>] \
[--iceberg-schema=<placeholder-text>] \
[--iceberg-snapshot-id=<placeholder-text>] \ 
[--iceberg-table=<placeholder-text>] \
[--iceberg-warehouse=<placeholder-text>] \ 
[--instance-id=<instance-id>] \
[--is-local-ingestion=true|false] \ 
[--job-id=<job-id>] \ 
[--num-executors=<placeholder-text>] \ 
[--partition-by=<placeholder-text>] \
[--schema=<placeholder-text>] \
[--source-data-files=<source-data-files>] \
[--source-database=<source-database>] \
[--source-file-type=<source-file-type>] \ 
[--source-iceberg-table=<source-iceberg-table>] \
[--sync-status=<sync-status>] \
[--target-write-mode=<target-write-mode>] \
[--transformation-schema=<transformation-schema>] \
[--user-name=<user-name>] \
[--validate-csv-header=true|false] \
[--validate-db-server-cert=true|false]

Options

Table 1: Command options
Option Description
--bucket-access-key Bucket access key.
Status
Optional.
Syntax
--bucket-access-key=<placeholder-text>
Input
string
Default value
No default.
--bucket-details Provide adhoc bucket details if the bucket is not already registered.
Status
Optional.
Syntax
--bucket-details=<placeholder-text>
Input
string
Default value
No default.
--bucket-endpoint Bucket endpoint.
Status
Optional.
Syntax
--bucket-endpoint=<placeholder-text>
Input
string
Default value
No default.
--bucket-name Bucket name.
Status
Optional.
Syntax
--bucket-name=<placeholder-text>
Input
string
Default value
No default.
--bucket-region Bucket region.
Status
Optional.
Syntax
--bucket-region=<placeholder-text>
Input
string
Default value
No default.
--bucket-secret-key Bucket secret key.
Status
Optional.
Syntax
--bucket-secret-key=<placeholder-text>
Input
string
Default value
No default.
--bucket-type Bucket type.
Status
Optional.
Syntax
--bucket-type=<placeholder-text>
Input
string
Default value
No default.
Valid values
amazon_s3, aws_s3, minio, ibm_cos, ibm_ceph, adls_gen1, adls_gen2, google_cs, ibm_storage_scale, ozone
--csv-encoding Encoding used in CSV file.
Status
Optional.
Syntax
--csv-encoding=<placeholder-text>
Input
string
Default value
The default value is utf-8.
--csv-escape-character Escape character of CSV file.
Status
Optional.
Syntax
--csv-escape-character=<placeholder-text>
Input
string
Default value
The default value is \.
--csv-field-delimiter Field delimiter of CSV file.
Status
Optional.
Syntax
--csv-field-delimiter=<placeholder-text>
Input
string
Default value
The default value is , .
--csv-header Identify if header exists in CSV file.
Status
Optional.
Syntax
--csv-header=<placeholder-text>
Input
boolean
Default value
The default value is true.
--csv-line-delimiter Line delimiter of CSV file.
Status
Optional.
Syntax
--csv-line-delimiter=<placeholder-text>
Input
string
Default value
The default value is \n.
--csv-property Provide CSV file properties.
Status
Optional.
Syntax
--csv-property=<placeholder-text>
Input
string
Default value
No default.
--database-auth-value --database-auth-value
Status
Optional.
Syntax
--database-auth-value=<placeholder-text>
Input
string
Default value
No default.
--database-cert-extension --database-cert-extension
Status
Optional.
Syntax
--database-cert-extension=<placeholder-text>
Input
string
Default value
No default.
Valid values
pem, crt, cert, cer
--database-certificate Certificate content as string.
Status
Optional.
Syntax
--database-certificate=<placeholder-text>
Input
string
Default value
No default.
--database-connection-mode Connection mode for Oracle.
Status
Optional.
Syntax
--database-connection-mode=<placeholder-text>
Input
string
Default value
No default.
Valid values
sid, service_name
--database-connection-mode-value Connection mode value for Oracle.
Status
Optional.
Syntax
--database-connection-mode-value=<placeholder-text>
Input
string
Default value
No default.
--database-host Database hostname.
Status
Optional.
Syntax
--database-host=<placeholder-text>
Input
string
Default value
No default.
--database-id Database ID.
Status
Optional.
Syntax
--database-id=<placeholder-text>
Input
string
Default value
No default.
--database-isssl Port is SSL enabled.
Status
Optional.
Syntax
--database-isssl=<placeholder-text>
Input
boolean
Default value
The default is false.
--database-name Database name.
Status
Optional.
Syntax
--database-name=<placeholder-text>
Input
string
Default value
No default.
--database-password Database password.
Status
Optional.
Syntax
--database-password=<placeholder-text>
Input
string
Default value
No default.
--database-port Database port.
Status
Optional.
Syntax
--database-port=<placeholder-text>
Input
string
Default value
No default.
--database-schema Database source schema name.
Status
Optional.
Syntax
--database-schema=<placeholder-text>
Input
string
Default value
No default.
--database-table Database source table name.
Status
Optional.
Syntax
--database-table=<placeholder-text>
Input
string
Default value
No default.
--database-type Database type.
Status
Optional.
Syntax
--database-type=<placeholder-text>
Input
string
Default value
No default.
Valid values
Database types include the following examples: db2, netezza, oracle, postgresql, snowflake, sqlserver, and mariadb.
--database-user-id Database username.
Status
Optional.
Syntax
--database-user-id=<placeholder-text>
Input
string
Default value
No default.
--database-warehouse Warehouse name for Snowflake.
Status
Optional.
Syntax
--database-warehouse=<placeholder-text>
Input
string
Default value
No default.
--driver-cores Driver core(s) configuration for Spark engine.
Status
Optional.
Syntax
--driver-cores=<placeholder-text>
Input
int64
Default value
No default.
--driver-memory Driver memory configuration (in GB) for Spark engine. .
Status
Optional.
Syntax
--driver-memory=<placeholder-text>
Input
string
Default value
The default value is 2G.
--engine-id Spark Engine ID for Spark ingestion or "lite-ingestion" for faster ingestion.
Status
Required.
Syntax
--engine-id=<engine-id>
Input
string
Default value
No default.
--execute-config Provide Spark engine configurations.
Status
Optional.
Syntax
--execute-config=<placeholder-text>
Input
string
Default value
No default.
Valid values
format: {"executor_memory":"", "num_executors":"", "driver_memory":"", "driver_cores":"", "executor_cores":""}
--executor-cores Executor core(s) configuration for Spark engine.
Status
Optional.
Syntax
--executor-cores=<placeholder-text>
Input
int64
Default value
No default.
--executor-memory Executor core(s) configuration for Spark engine.
Status
Optional.
Syntax
--executor-memory=<placeholder-text>
Input
int64
Default value
No default.
--iceberg-catalog Source iceberg catalog name.
Status
Optional.
Syntax
--iceberg-catalog=<placeholder-text>
Input
string
Default value
No default.
--iceberg-schema Source iceberg schema name.
Status
Optional.
Syntax
--iceberg-schema=<placeholder-text>
Input
string
Default value
No default.
--iceberg-snapshot-id Source iceberg snapshot ID.
Status
Optional.
Syntax
--iceberg-snapshot-id=<placeholder-text>
Input
string
Default value
No default.
--iceberg-table Source iceberg table name.
Status
Optional.
Syntax
--iceberg-table=<placeholder-text>
Input
string
Default value
No default.
--iceberg-warehouse Source iceberg warehouse name.
Status
Optional.
Syntax
--iceberg-warehouse=<placeholder-text>
Input
string
Default value
No default.
--instance-id watsonx.data instance ID for IBM Software Hub or CRN for SaaS.
Status
Optional.
Syntax
--instance-id=<instance-id>
Input
string
Default value
No default.
--is-local-ingestion If value is true, ingestion mode is local. If value is false, ingestion mode is remote.
Status
Optional.
Syntax
--is-local-ingestion=true|false
Input
boolean
Default value
false
--job-id Ingestion job ID is used to differentiate between jobs. If job ID is not provided, a unique random job ID will be generated by default.
Status
Optional.
Syntax
--job-id=<job-id>
Input
string
Default value
No default.
--num-executors Number of executors to assign for Spark engine.
Status
Optional.
Syntax
--num-executors=<placeholder-text>
Input
integer
Default value
No default.
--partition-by Partition by expression of the target table.
Status
Optional.
Syntax
--partition-by=<placeholder-text>
Input
string
Default value
No default.
--schema Schema definitions for the source data.
Status
Optional.
Syntax
--schema=<placeholder-text>
Input
string
Default value
No default.
Valid values
format: {"headerName":"", "type":"", "fieldId":""}
--source-data-files Comma separated source file(s) or directory path.
Status
Optional.
Syntax
--source-data-files=<source-data-files>
Input
string
Default value
No default.
--source-database Provide adhoc source database details if the database is not already registered.
Status
Optional.
Syntax
--source-database=<source-database>
Input
string
Default value
No default.
Valid values
 format: {"authentication_value":"", 
"database_name":"", 
"connection_mode_value":"", 
"certificate":"", 
"schema_name":"", 
"table_name":"", 
"userId":"", 
"database_type":"", 
"password":"", 
"warehouse_name":"", 
"cert_extension":"", 
"database_id":"", 
"port":"", 
"validate_server_certificate":"", 
"connection_mode":"", 
"host":"", 
"isssl":""}
--source-file-type Source file types.
Status
Optional.
Syntax
--source-file-type=<source-file-type>
Input
string
Default value
No default.
Valid values
Source file types such as parquet, csv, json, orc, and avro.
--source-iceberg-table Provide source iceberg details.
Status
Optional.
Syntax
--source-iceberg-table=<source-iceberg-table>
Input
string
Default value
No default.
Valid values
Example format:
{"catalog_name":"", 
"warehouse_name":"", 
"snapshot_id":"", 
"schema_name":"", 
"table_name":""}
--sync-status Retrieve status synchronously as the ingestion job is sent.
Status
Optional.
Syntax
--sync-status=<sync-status>
Input
boolean
Default value
false
--target-table Target table name.
Status
Required.
Syntax
--target-table=<target-table>
Input
string
Default value
No default.
--target-write-mode Determines if the spark job should append to target or overwrite it.
Status
Optional.
Syntax
--target-write-mode=<target-write-mode>
Input
string
Default value
The default value is append.
Valid values
Allowable values include append and overwrite.
--transformation-schema Transformation schema for altering the table schema.
Status
Optional.
Syntax
--transformation-schema=<transformation-schema>
Input
string
Default value
No default.
--user-name Specify a username that corresponds to a project member.
Status
Optional.
Syntax
--user-name=<user-name>
Input
string
Default value
No default.
--validate-csv-header Validate CSV header if the target table exist.
Status
Optional.
Syntax
--validate-csv-header=true|false
Input
boolean
Default value
false
--validate-db-server-cert Verify provided certificate.
Status
Optional.
Syntax
--validate-db-server-cert=true|false
Input
boolean
Default value
false

Examples

Remote ingestion:

cpd-cli wx-data ingestion create \
--engine-id  spark637 \
--target-table sample_catalog.sample_schema.sample_tabl \
--csv-property '{"header": true, "encoding": "utf-8", "escape_character": "\\", "line_delimiter": "\n", "field_delimiter": ","}' \
--instance-id 1735472262311515 \
--source-data-files s3://sample-bucket/sample_file.csv

Lite ingestion:

cpd-cli wx-data ingestion create \
--engine-id lite-ingestion \
--bucket-name transient_bucket_name \
--instance-id 1735472262311515 \
--source-data-files /path_to_file/file.orc \
--target-table sample_catalog.sample_schema.sample_table

Database ingestion:

cpd-cli wx-data ingestion create \
--engine-id spark266 \
--target-table sample_catalog.sample_schema.sample_table \
--database-id sample_database_id \
--database-schema sample_database_name \
--database-table sample_database_table \
--instance-id 1735472262311515

Iceberg ingestion:

cpd-cli wx-data ingestion create \
--engine-id spark266 \
--target-table sample_catalog.sample_schema.sample_table \
--bucket-name iceberg-data
--iceberg-catalog sample_iceberg_catalog \
--iceberg-schema sample_iceberg_schema \
--iceberg-snapshot-id 7823318841638214979 \
--iceberg-table sample_iceberg_table  \
--iceberg-warehouse sample_iceberg_warehouse \
--instance-id 1735472262311515