Mapping to Cloudant

The latest CDC Replication Engine for InfoSphere® DataStage® supports specifying delivery to Cloudant®.

Understanding the workflow

Replication process begins once a refresh or mirroring operation begins. CDC Replication starts writing change information to memory for only those tables in the subscription for which there are changes. Once the batch size threshold limits are met (the batch limit can be set via the system parameter: global_net_effect_stage_batch_size), CDC Replication applies data to Cloudant.

A source row is represented in Cloudant as a document containing key-value pairs with the source column names as the keys and the values of those columns for the row as the values except when a parent table has been selected for the mapping.

Rather than being represented as a document, a source row for a table with a parent table will be represented as an element of a JSON array of JSON objects in a document corresponding to the parent table. The JSON object for the child row will contain key-value pairs corresponding to the column names and row values. The containing Cloudant document will have a key-value pair with the source table name as the key and the JSON array of JSON objects as the value. The elements of the JSON array will be the JSON objects corresponding to source rows with the column values matching the primary key of the parent row.

In addition to the column key-value pairs, top-level documents will have a key named DATAWORKS_DOCUMENT_TYPE with the source table name as the value and both documents and child objects will have keys DATAWORKS_AUDIT_TIMESTAMP and DATAWORKS_AUDIT_USER, corresponding to the journal control fields &TIMSTAMP and &USER respectively.