Synchronizing Data Between Multiple Resources

As stated above, data synchronization keeps equivalent data across multiple systems consistent. Equivalent data in resources does not necessarily use the same layout or structure. The data structure is based on the requirements of the resource. The following example shows different data structures for customer data that a CRM system and a Billing system might use:

Customer ID    
Customer Name    
       First    
       Surname    
Customer Address    
       Line1    
       Line2    
       City    
       State    
       ZipCode    
       Country    
Customer Payment Information    
Customer Account

Account ID    
Account Type    
Billing Account Owner    
       Last    
       First    
Billing Address    
       Number    
       Street    
       AptNumber    
       CityOrTown    
       State    
       Code    
       Country    
Billing Preferences

Data in an application contains a key value that uniquely identifies an object within the application. In the example above, the key value that uniquely identifies a customer within the CRM system is the Customer ID; similarly, the key value in the Billing system is the Account ID. The key value in a specific application is referred to as that application’s native ID. In other words, the native ID for the CRM system is the Customer ID, and the native ID for the Billing system is the Account ID.

Source resources send documents to notify other resources of data changes. You set up processing to map the data from these notification documents into a canonical document. A canonical document is a document with a neutral structure, and it encompasses all information that resources will require to incorporate data changes. Use of canonical documents:

• Simplifies data synchronization between multiple resources by eliminating the need for every resource to understand the document structure of every other resource. Resources need only include logic for mapping to and from the canonical document. Without a canonical document, each resource needs to understand the document structures of the other resources. If a change is made to the document structure of one of the resources, logic for data synchronization would need to change in all resources.

  For example, consider keeping data synchronized between three resources: a CRM system, Billing system, and Order Management system. Because the systems map directly to and from each others native structures, a change in one resource’s structure affects all resources. If the address structure was changed for the CRM system, you would need to update data synchronization logic in all resources.

  

  When you use a canonical document, you limit the number of data synchronization logic changes that you need to make. Using the same example where the address structure changes for the CRM system, you would only need to update data synchronization logic in the CRM system.

  
• Makes adding new resources to the data synchronization easier. With canonical documents, you only need to add data synchronization logic to the new resource to use the canonical document. Without a canonical document, you would need to update the data synchronization logic of all resources so they understand the structure of the newly added resource, and the new resource would need to understand the document structures of all the existing resources.

Key cross-referencing allows you to use a common key (i.e., the canonical ID) to build relationships between equivalent business objects from different resources. You build the relationships by mapping key values (i.e., the native IDs) from the resources to a common canonical ID. You maintain these relationships in the cross-reference database component (if using an external RDBMS) or the cross-reference table (if using the embedded internal database). For simplicity, in this chapter the term cross-reference table is used to encompass both. For information about configuring the cross-reference table, see Configuring Integration Server for Key Cross-Reference and Echo Suppression.

The cross-reference table includes the fields described in the following table:

For example, to synchronize data between a CRM system, Billing system, and Order Management system, you might have the following rows in the cross-reference table:

One other feature that Integration Server provides for data synchronization is echo suppression, which is also called latching. Echo suppression (or latching) is the process of preventing circular updating from occurring. Circular updating can occur when performing n-way synchronizations (data synchronizations where every resource can be a source and a target as well).

Circular updating occurs when the source subscribes to the canonical document that it publishes as illustrated in the diagram below.

In addition to the source immediately receiving the canonical that it formed, it can also receive the canonical document many more times because other targets build and publish the canonical document after making the data change that the source initiated. See below for an illustration of this circular updating.

To avoid the circular updating, Integration Server provides you with the following tools to perform echo suppression:

• The isLatchClosed field in the cross-reference table that you use to keep track of whether a resource has already made a data change or not.
• Built-in services that you use to manipulate the isLatchClosed field in the cross‑reference table to:
  • Determine the value of the isLatchClosed field, and
  • Set the value of the isLatchClosed column.

When you use an adapter to manage the source resource, configure the adapter to send an adapter notification when a change occurs on the resource. For information about how to configure the adapter, see the documentation for the adapter.

If the adapter does not create an publishable document type for the adapter notification, use Software AG Designer to define an publishable document type that defines the structure of the adapter notification.

Based on the adapter, the adapter does one of the following to send the adapter notification:

• Publishes the adapter notification. A trigger on the source Integration Server subscribes to this publishable document type. When the trigger receives a document that matches the publishable document type for the adapter notification, it invokes the trigger service that builds the canonical document.
• Directly invokes the service that builds the canonical document. When the service is directly invoked, the adapter notification is sent to the service as input.

For more information about creating the service that builds the canonical document, see Setting Up Key Cross-Referencing in the Source Integration Server.

When you develop your own logic to interact with the source resource, the logic should include sending a document when a data change occurs within the resource. You define the document fields that you require for the notification. Be sure to include a field for the native ID to identify the changed object on the source.

After determining the fields that you need in the source native document (i.e., the notification), use Software AG Designer to define an IS document type for the native document. The logic you create to interact with the source resource can do one of the following to send the source native document:

• Publish the source native document. If your logic publishes the source’s native document, define a publishable document type for the source’s native document. A trigger on the source Integration Server subscribes to this publishable document type. When the trigger receives a document that matches the publishable document type for the source’s native document, it invokes the trigger service that builds the canonical document.
• Directly invoke the service that builds the canonical document, When the service is directly invoked, the source’s native document is sent to the service as input. If your logic passes the native document to the service as input, the IS document type does not need to be publishable.

For more information about creating the service that your logic should invoke, see Setting Up Key Cross-Referencing in the Source Integration Server.

To set up key cross‑referencing in the source Integration Server:

• Create a trigger to subscribe to the source resource’s notification document, if applicable. You only need to define a trigger that subscribes to the notification document if the source resource publishes the notification document (i.e., either an adapter notification or other native document). If the source directly passes the notification document to a service as input, you do not need to define a trigger. If you need to define a trigger, define the trigger that:
  • Subscribes to the publishable document type that defines the notification document (i.e., either an adapter notification or other native document).
  • Defines the trigger service to be the service that builds the canonical document from the notification document.

  For more information, see webMethods Service Development Help .

• Create the trigger service that builds the canonical document and publishes the canonical document to propagate the data change to all target resources. You need to create a service that puts the data change information into a neutral format that all targets understand. The neutral format is the canonical document.

  The service builds the canonical document by mapping information from the notification document to the canonical document. To obtain the canonical ID for the canonical document, the service uses the built‑in key cross‑referencing services pub.synchronization.xref:getCanonicalKey and/or pub.synchronization.xref:createXReference, as shown in the sample logic below. After forming the canonical document, your service publishes the canonical document.

  

  Step	Description
  1	Determine whether there is already a canonical ID. Invoke the pub.synchronization.xref:getCanonicalKey service to locate a row in the cross‑reference table for the source object. If the row already exists, a canonical ID already exists for the source object. Pass the getCanonicalKey service the following inputs that identify the source object:
  	In this input variable...	Specify...
  	appId	The identification of the application (e.g., CRM system).
  	objectId	The string that you assigned to identify the object (e.g., Account). This string is referred to as the object ID.
  	nativeId	The native ID from the notification document (e.g., adapter notification), which was received as input to your service.
  	If the getCanonicalKey service finds a row in the cross‑reference table that matches the input information, it returns the value of the canonical ID in the canonicalKey output variable. If no row is found, the value of the canonicalKey output variable is blank (i.e., an empty string). For more information about the getCanonicalKey service, see the pub.synchronization.xref:getCanonicalKey.
  2	Split logic based on whether the canonical ID already exists. Use a BRANCH flow step to split the logic. Set the Switch property of the BRANCH flow step to canonicalKey.
  3	Build a sequence of steps to execute when the canonical ID does not already exist. Under the BRANCH flow step is a single sequence of steps that should be executed only if a canonical ID was not found. Note that the Label property for the SEQUENCE flow step is set to blank. At run time, the server matches the value of the canonicalKey variable to the Label field to determine whether to execute the sequence. Because the canonicalKey variable is set to blank (i.e., an empty string), the label field must also be blank. Important: Do not use $null for the Label property. An empty string is not considered a null.
  4	If there is no canonical ID, define one. If a row for the source object is not found in the cross‑reference table, there is no canonical ID for the source object. Define a canonical ID by adding a row to the cross‑reference table to cross‑reference the source native ID with a canonical ID. You add the row by invoking the pub.synchronization.xref:createXReference service. Pass the createXReference service the following:
  	In this input variable...	Specify...
  	appId	The identification of the application (e.g., CRM system).
  	objectId	The object type (e.g., Account).
  	nativeId	The native ID from the notification document (e.g., adapter notification), which was received as input to your service.
  	canonicalKey	(optional) The value you want to assign the canonical ID.
  	If you do not specify a value for the canonicalKey input variable, the createXReference service generates a canonical ID for you. For more information about the createXReference service, see the pub.synchronization.xref:createXReference.
  5	Build the canonical document. Map fields from the notification document (e.g., adapter notification) to the fields of the canonical document. Make sure you map the canonical ID generated in the last step to the canonical ID field of the canonical document. The notification document has the structure that you previously defined with a the publishable document type. See Defining How a Source Resource Sends Notification of a Data Change. Similarly, the canonical document has the structure that you previously defined with a publishable document type. See Defining the Structure of the Canonical Document. Note: Although this sample logic shows only a single MAP flow step, you might need to use additional flow steps or possibly create a separate service to build the canonical document.
  6	Publish the canonical document. After the service has formed the canonical document, invoke the pub.publish:publish service to publish the canonical document to the messaging provider.

The echo suppression logic in a notification service determines whether a latch is open or closed before it attempts to build and publish the canonical document.

• If the latch is open, the resource is the source of the data change. In this case, the notification service on the source Integration Server builds the canonical document and publishes it. The notification service should include logic that closes the latch to prevent a circular update.
• If the latch is closed, the resource has already made the data change. In this case, the notification service does not need to build the canonical document to notify resources about the data change because the notification service on the source Integration Server has already done so. The notification service should simply re‑open the latch and terminate processing.

The following diagram highlights the part of data synchronization that this section addresses.

The update trigger service receives the canonical document that describes a data change. The echo suppression logic in an update service determines whether a latch is open or closed before it attempts to use the information in the canonical document to update a resource with the data change.

• If the latch is open, the data change has not yet been made to the resource. In this case, the update service builds and sends the native document that informs the target resource of the data change. The update service closes the latch to prevent a circular update.
• If the latch is closed, the resource was the source of the data change and has already made the data change. In this case, the update service does not need build and send the native document. The update service should simply re‑open the latch and terminate processing.

The following diagram highlights the part of data synchronization that this section addresses.