IBM StreamSets
Using the IBM StreamSets, you can transform and publish data from the IBM Product Master to Snowflake and Magento.
Key concepts
Following are the key concepts that you must know for this integration.
| Term | Description |
|---|---|
| Control Hub | IBM StreamSets Control Hub provides a graphical user interface (GU) for managing the pipeline and the execution components. |
| Data Collector (Pipeline) | IBM StreamSets Data Collector is a pipeline that transmits and transforms the raw data from the origin to the destination. |
| IBM Product Master | This is the origin and first node in the pipeline. This contains the data that comes to the
IBM StreamSets from Product Master application by using
the REST APIs. For more information, see the REST Service topic in the IBM StreamSets documentation. |
| Select Application ID | This is the second node in the pipeline. This is the Stream Selector that decides where to
send the data from the first node. For more information, see the Stream Selector topic in the IBM StreamSets documentation. |
| Processors | These processors are used for data transformation and storage on the required
destination. For more information, see the Processors topic in the IBM StreamSets documentation. |
| Snowflake destination | The Snowflake destination writes data to one or more tables in a Snowflake database. For more information, see the Snowflake topic in the IBM StreamSets documentation. |
| Magento destination | The Magento destination writes data to the Magento. |
IBM StreamSets-Product Master integration pipeline
The following image describes a basic flow.
Sample Request Payload (Request Body)
The IBM StreamSets uses a Request Payload to transform the Product Master data and then publish the transformed data to the respective .
Following is an example of a sample payload file.
{
"primaryKeyNodeName": "<Spec Name>/<Attribute Name>",
"displayNodeName": "<Spec Name>/<Attribute Name>",
"batchId": "generated batch id ",
"entryInfoList": [
{
"primaryKey": "<item Primary key>",
"entryData": {
"<Spec Name>/<Attribute Name>": "<Attribute value>",
"<Spec Name>/<Attribute Name>": "<Attribute value>",
"<Spec Name>/<Attribute Name>": "<Attribute value>",
"<Spec Name>/<Attribute Name>": "<Attribute value>",
"<Secondary Spec Name>/<Attribute Name>": "<Attribute value>",
"<Secondary Spec Name>/<Attribute Name>": "<Attribute value>",
"<Secondary Spec Name>/<Attribute Name>": "<Attribute value>"
},
"parentCategories": [
"<Hierarchy Name>/<Hierarchy Path>"
]
}
],
"containerName": "<Catalog Name>",
"totalCount": "<entries count>"
}