Bulk operation methods

Bulk operation methods are used to run bulk data load and update jobs on the Master Data Connect system.

Start an operation to bulk load data into the graph

POST /mdm/cache/v0/bulk_load

The bulk load operation requires users to have the application role of Writer.

Considerations:
  • The following sample data sets are available. These are the only acceptable values for directory_ref:
    • sample_contract_small
    • sample_consent_small
    • sample_contract
    • sample_consent
  • The job id is a random, universally unique identifier that is also appended to each job driver pod.
  • To run a sample bulk load, provide the type as sample and the provide a directory_ref identifying which sample data set to load.
    Tip: directory_path is not required when the type is set to sample. If provided it will be ignored.
  • To run a dfs bulk load, provide the type as dfs and a directory_path pointing to the location of the load data.
    Tip: directory_ref is not required when the type is set to dfs. If provided it will be ignored.
    Load data directories are expected to adhere to the following format:
    record.properties
relationship.properties
record
--[record data files]
relationship
--[relationship data files]
Request
Request body:
BulkLoadRequest
Valid JSON defining the data source and parameters for the bulk load job.
data_source
object:
type
string
Allowable values:
  • dfs
  • sample
directory_path
string
directory_ref
string
Allowable values:
  • sample_contract_small
  • sample_consent_small
  • sample_contract
  • sample_consent
Response
Status code Description
202 The bulk load job was successfully started.
400 Input validation failed.
401 Problem processing request. The user is not authenticated.
403 Problem processing request. The user is not authorized to perform the request.
500 Problem performing bulk load. An internal error occurred while attempting to perform the operation.
503 Problem performing bulk load. A bulk load process is already running.

Start an operation to perform a bulk update to the graph

POST /mdm/cache/v0/bulk_update

The bulk load operation requires users to have the application role of Manager.

Considerations:
  • This method performs all relationship deletions first, then record deletions.
  • After the deletions are completed, all record insertions/updates are performed next, followed by relationship insertions/updates.
  • Any element found in both deletions and insertions/updates is treated as a deletion, and is removed from the insertions/updates list before processing.
  • Any other case of a duplicated element will cause the update to fail.
  • If the ignore_redundant_updates flag is set to true, any update with a timestamp that is not newer than the existing timestamp for that element will not be applied, but it will not cause the bulk update transaction to fail. If the flag is set to false, invalid timestamps will cause failure.
  • A failed update will cause all changes performed by the bulk update transaction to be reverted back to the original graph state.
Request
Query parameters:
ignore_redundant_updates
Boolean. Defines whether to ignore updates that fail due to missing or invalid record_last_updated timestamps.
Request body:
SyncUpdateRequest
Valid JSON defining the elements to be inserted, updated or deleted on the graph.
upserts
object
type
string
id
string
attributes
object
any property
object
type_name
string
entity_count
int64
relationships
Relationship[]
type
string
id
string
attributes
object
any property
object
type_name
string
from_record_id
string
from_record_source
string
from_record_type
string
to_record_id
string
to_record_source
string
to_record_type
string
deletions
object
records
Record[]
type
string
id
string
attributes
object
any property
object
type_name
string
entity_count
int64
relationships
Relationship[]
type
string
id
string
attributes
object
any property
object
type_name
string
from_record_id
string
from_record_source
string
from_record_type
string
to_record_id
string
to_record_source
string
to_record_type
string
Response
Status code Description
204 The bulk update job was successfully processed.
400 Input validation failed.
401 Problem processing request. The user is not authenticated.
403 Problem processing request. The user is not authorized to perform the request.
500 Problem performing bulk load. An internal error occurred while attempting to perform the operation.