Using Suspect Duplicate Processing (SDP)

Suspect Duplicate Processing involves identifying possible suspects, matching, and if applicable merging data from an Operational catalog to the Master catalog. When an item is imported or updated into the Operational catalog, the item is compared against the existing items of the Master catalog, and duplicate details, if any are displayed in the Suspect Duplicate Processing tab. Then, the item can be processed further based on the user selection.

As an example, consider a data model where Product Master is used to perform master data management of products. After a product is introduced or updated in SDP Collaboration Area or Operational catalog, SDP engine by using OpenSearch compares attributes of incoming product against the existing products present in the Master catalog. Based on the product attributes like Product Identifier, Name, Description, the SDP engine displays a list of possible suspects with duplication score. You can review and compare all the duplicate suspects with this incoming product through attributes merge table. Attributes merge table is displayed only after selection of any one of the possible suspects. After reviewing all the suspected duplicate products, you can mark the incoming product as a Match or No Match.

SDP might have the following possible scenarios:

Scenario: Confident that the two products are the same
If all or important attributes of current incoming product are exactly same as those of an existing master product and you are confident that the incoming product is duplicate, then select the existing master product and click Match to mark incoming product as a duplicate of the master product. Once a product is marked as a Match (duplicate product), then the product remains in the Operational Catalog and is not created in the Master Catalog.
Scenario: Unsure if the two products are the same
If important attributes of current incoming product are partially matching with those of an existing master product and you are confident that the incoming product is duplicate, then select the existing master product and click Match to mark incoming product as a duplicate of the respective master product. Also, if you are confident that the incoming product attributes are more accurate than those of the existing master product, you can merge attributes from the incoming product to the existing master product by selecting individual attributes and clicking Merge.
Scenario: Confident that the two products are not the same
If only one or a few low-priority attributes are matching, and you are confident that two products are not the same, then mark the incoming product as a No Match. After you select No Match, the product is created in the master catalog.

Prerequisites

Ensure that you have completed the following tasks:
  • Deploy and configure the latest Admin UI and Persona-based UI versions.
  • Enable the Free text search feature and start the Free text search services.
  • Identify, create, and index all catalogs with OpenSearch.
  • Deploy the sdp-ext.jar file that is located at $TOP/mdmui/libs/mdm/sdpExtensions and update the classpath for the Admin UI and Persona-based UI. Get the list of master catalogs and users performing the SDP operations.
  • Configure SDP for catalogs. For more information, see Running SDP automation script.

SDP terms

Following terms are used in the SDP:
Golden copy
The item exists in the Master catalog.
Match
An item in the Operational catalog is a duplicate of an item in the Master catalog.
No match
The item in the Operational catalog that is not a duplicate.

Fix Pack 12 enhancements

  • Auto SDP -
    • Self-comparison does not happen.
    • Merged date gets updated.
  • Data explorer page: Operational catalog -
    • Existing operational catalogs - Multiple reference fields continue to be visible on the Data explorer page.
    • For the catalogs on which the SDP script will be run after the release of the IBM Product Master Fix Pack 12.0, multiple reference fields will not be visible on the Data explorer page.
  • New reference attribute collection - In Admin UI, for each catalog on which a script is run, a new <Catalog Name> + Reference Attribute Collection reference attribute collection gets created.

    This new attribute collection gets added to each step in the workflow (Product enrichment, Auto SDP, and SDP step) along with the <Catalog Name> + SDP Attribute Collection.

    For existing SDP attribute collection configurations in the SDP workflow, a new <Catalog Name> + Reference Attribute Collection reference attribute collection gets appended to each step in the workflow.

  • Reference attribute collection - The reference attribute collection is not updated for new catalogs on which the script is triggered and instead new <Catalog Name> + Reference Attribute Collection gets created.