Bulk loading items

Bulk loading items feature enables any third-party integration to perform loading of items into the IBM Product Master catalog efficiently.

Fix Pack 7 and later

There are two integration APIs available to fetch schema and load items, and the APIs are compliant with the standard JSON schema. Bulk load is done by scheduled jobs thus, keeping the APIs separate from backend processing.

Prerequisites

  • Import mdmce-env.zip file. After the file imported successfully, verify that the data model mentioned in the Importing data model topic for the Bulk loading items feature is present.
  • The common.properties file has data_import_queue properly configured. For more information, see common.properties file parameters.

Procedure

  1. Get the spec schema by using Get Schema API.
    Usage
    Get Spec Schema for catalog.
    Method
    GET
    URL
    https://productmaster.<host>.com/api/v1/integrations/catalogs/<catalogname>/schema?category=<categoryname>
    Where,
    • catalogname - Specifies the name of the catalog.
    • category name - Specifies the name of the category whose secondary spec schema needs to be returned in response. It is an optional query parameter. If not specified, schema for all the secondary specs for all categories from all hierarchies that are associated with the catalog gets returned in response.
    Table 1. Data types and standard JSON schema types
    Data type Standard JSON schema type
    CURRENCY Number
    NUMBER
    NUMBER_ENUMERATION
    FLAG Boolean
    INTEGER Integer
    SEQUENCE
    Other data types String
    For more information on response body, see Integrations REST APIs - Catalog.
  2. Push the item data JSON to queue by using bulk load API.
    Usage
    Load bulk items to queue to import to the catalog.
    Method
    POST
    URL
    https://productmaster.<host>.com/api/v1/integrations/bulkload
    Request and response body
    • Primary spec of catalog in which the item needs to be imported must have an attribute, which is unique, indexed, and mandatory. Attribute path of this attribute should be mentioned against identifierAttributeName and the attribute path value should be mentioned against identifierAttributeValue in the Request body.
    • Following table represents the supported and non-supported attribute type for an attribute mentioned against identifierAttributeName in the request body.
      Supported Attribute Type Non-Supported Attribute Type
      Currency Binary
      Date Flag
      Integer Image
      Image URL Lookup Table
      Number Password
      Number Enumeration Relationship
      Rich Text Thumbnail Image
      Sequence Time Zone
      String Linked
      String Enumeration
      Thumbnail Image URL
      URL
    For more information on request and response body, see Integrations REST APIs - Catalog.
    Deduplication of items
    If request JSON contains multiple data JSON objects for the same item,
    • Multiple item data JSON objects get merged into a single JSON such that attributes values from the last JSON entry overwrites all the previous entries present in the request body.
    • Category mappings from multiple item data JSON objects get merged.
    Survivorship
    In the Survivorship Configuration Lookup table, each attribute is mapped to a trusted source system from which application can accept an update.
    Source for each item in the Request JSON gets checked against the trusted source system from the Survivorship Configuration Lookup table to decide whether to accept the update on an individual attribute basis.
    Survivorship Configuration Lookup table details
    Name: Survivorship Configuration
    Attributes:
    • Source: Name of the source for item data.
    • Attributes: List of comma-separated leaf attribute (attributes that have no child attribute) full path
      Example
      Product Spec/Product Name, Product Spec/Price, Product Spec/Delivery Address/City
    Mandatory attributes should be present in Attribute’ against the specific catalog in the Lookup table, else item save fails.
    Note: If the Survivorship Configuration Lookup table is empty, then the survivorship check is skipped.
  3. Configure and trigger Bulk Load Job Report’to fetch item data JSON from queue and load items the catalog.
    Usage
    Fetch item data JSON from queue and load items in a catalog.
    Configuration
    Following table lists the inputs that are required for configuration.
    Name Type Description Is Mandatory? Default Value
    Container Name String Name of the catalog to which items need to be imported. Yes NA
    Batch Size String Number of items to be committed to the database in one go. No 500
    Poll timeout (secs) String Timeout for queue polling. (Job waits for messages to be available on the queue until poll timeout expires.) No 3600 secs
    Run the report job to load items into the configured catalog.
    • If an item exists in the catalog matching identifierAttributeName & identifierAttributeValue, then existing item gets updated. Category is mapped (according to mappings from the item data JSON) and attribute is set (according to attributes from item data JSON.)
    • Otherwise, new item is created in the catalog. Category is mapped (according to mappings from the item data JSON) and attribute is set (according to attributes from item data JSON.)
    • Loading existing items into the catalog:
      • Multi-occurring attributes: All the occurrences are replaced with incoming data. This is true for:
        • Multi-occurring attribute of simple type (say String),
        • Multi-occurring grouping attribute,
        • Grouping attribute containing multi-occurring attribute.
      • Category mappings:
        • If mappings has null or empty array (that is []) value in item data JSON, there is no change in existing category mapping of the item.
        • Existing category mapping for item is never deleted.
    • New item import: Ifmappings has null or empty array, item gets created under unassigned category.
    • Move item to collaboration step: If there is error while saving an item to a catalog, the item gets created or updated in collaboration area step configured against the catalog in the Bulk Load Error Configuration Lookup table.
      Bulk Load Error Configuration Lookup table details
      Name: Bulk Load Error Configuration
      Attributes:
      • Container Name
      • Collaboration Area Name
      • Collaboration Step Name
      Collaboration area step name, which is configured in the Lookup table should have:
      • Allow Import into Step flag enabled.
      • Allow Recategorization flag enabled.
      Note: If the Bulk Load Error Configuration Lookup table is empty, then this step is skipped.
    After all the items from queue are processed or Poll Timeout expires, report job gets completed and report.out file is generated (successful completion of job) containing summary of import.