preview

Search index process flow

In IBM Sterling® Order Management System, indexing of transactional data occurs automatically when Sterling Search Index is enabled.

Index build process

Whenever a transactional data is created or modified in Sterling™ Order Management System, the Sterling Search Index senses the need for indexing and sends a subset to outline the transactional data for indexing after the changes are committed in Sterling Order Management System.

The capability to index transactional data is provided for order and shipment entities. If Sterling Search Index is enabled, as part of the business transaction flow, whenever you create or modify an order or shipment, Sterling Order Management System checks whether any of the indexable attributes are modified during the transaction and holds the information in memory, and sends an outline with the indexable attributes when the corresponding business APIs or agents are committed.

The Sterling Search Index supports the PeriodicBatchIndexer service, which accumulates the data received from different business transactions and periodically performs a batch update on the index with the information collected up to that point in time. The periodicity time of batch updates is configurable through the yfs.ssi.batch.index.periodicity.secs property in the yfs.properties_ysc_ext property file, which is by default set to 3 seconds. Thus, any changes to the outline data are updated on the index asynchronously, but near real time.

Sterling Order Management System is heterogeneous in nature and consists of application server, agent server, and Search Index Server. These servers are deployed outside of Sterling Order Management System. As indexing is done asynchronously, there is always a possibility that the requests to the Search Index Server might not succeed, time out, lose track, and so on. Furthermore, by default an index update is triggered only once in 3 seconds. If a server is forcefully stopped, a set of such changes is not submitted to the Search Index Server. Therefore, for each update to the search index a record is inserted in the YFS_Awaiting_Index table with the corresponding EntityName and EntityKey. For example, during an order update a record is inserted in the YFS_Awaiting_Index table with EntityName=Order_Header and EntityKey=<value of OrderHeaderKey>. This record is present in the same database shard as the corresponding entity record.

Sterling Order Management System tracks every index update operation in the YFS_Awaiting_Index table. For each indexing attempt that is successful, after the index update operation, the corresponding record is removed from the YFS_Awaiting_Index table. If the index update does not succeed for some reason, for example if the Search Index Server is down or the communication to the Search Index Server does not succeed or times out, an exception is logged and the corresponding records in YFS_Awaiting_Index remains that are later processed by the SSI_Delayed_Sync agent.

The following diagram illustrates the asynchronous index build logic:

The following steps describes the index build process flow:
  1. An order is created or updated in Sterling Order Management System, which identifies that at least one of the attributes in the outline template has been modified.
  2. An outline document, which is used for updating the index is now created for the Order record. This document is used for the actual index update operation as explained in steps 5 and step 6. However, a record in the YFS_Awaiting_Index table is created at this stage to track this particular index update operation.
  3. The business API or agent commits.
  4. The document created in step 2 is sent to the Sterling Search Index accumulator along with a reference to the corresponding YFS_Awaiting_Index record.
  5. The Sterling Search Index accumulates such documents coming from different business transactions and periodically submits them to the Search Engine Client by using the PeriodicBatchIndexer service.
  6. The Search Engine Client submits these index updates to the Search Index Server.
  7. The response from Search Index Server is parsed, and for update operations that are successful, the corresponding records are removed from the YFS_Awaiting_Index table. For operations that failed, the corresponding YFS_Awaiting_Index records are retained.
Note:
  • A built-in integration with Order Search is available in Sterling Order Management System to support indexing and search capabilities for Order entity. In this integration, Order Search is deployed outside Sterling Order Management System and acts as the Search Index Server, which constitutes a search engine and an index repository.
  • By default, the built-in integration with Order Service does not sync very large orders. This behavior exists to prevent sync failures and performance degradation. A very large order pulls huge amount of data from Sterling Order Management System for every order update. The sync to Order Service is a batched process that includes orders modified during the periodic sync interval (3 seconds by defaults). When very large orders are included, they increase the payload and can exceed system limits, resulting in exceptions. These exceptions stop the sync process, which also stops the regular order sync. To avoid this, a very large order is not considered for syncing to Order Service by default.

    The PREVENT_INDEXING_OF_VERY_LARGE_ORDER rule is added to allow syncing of very large orders to Order Service. By default, this rule is set to 'Y', which prevents such orders from being synced. You may change the value to 'N' if your order outline document contains only minimal order line data and you want to evaluate the sync behavior. If you observe any performance degradation or sync failures, it is recommended to revert to the default setting. The value of this rule can be modified only by calling manageRule API.

    The orders with order lines greater than or equal to VERY_LARGE_ORDER_SIZE rule value are considered as very large orders.