Index synchronization concepts

When running correctly, indexes are synchronized. However, there are several scenarios where an index can be out of sync.

An index is said to be in a consistent state when it is a true representation of its corresponding entity’s database state. For example, the ‘Order’ index is consistent if it is up-to-date with the contents of the YFS_ORDER_HEADER and related tables. In such a case, the index is a true representation of the database state, since a query on the index will return the same set of records as the database tables will. This is critical for proper functioning of the search APIs. If an order is updated through an API, but if its corresponding index is not updated, the order may not be returned as part of a search API call because the search API primarily relies on the index to determine where to fetch the actual orders from.

Since index update operations are done asynchronously after a small delay, and because there is a possibility of failure of the operation when attempted, the search index may not be strictly consistent all the time. However, as long as the Sterling Search Index is enabled and active, every index-update operation will be attempted in near real-time, and failures will be notified and tracked through the YFS_Awaiting_Index table and attempted again later by the SSI_DELAYED_SYNC agent. Therefore, though not strictly consistent all the time, as long as the Sterling Search Index is enabled, the index can be assumed to be ‘in sync’ with the database for all practical purposes under steady use.

However, there are scenarios where the index may get considerably out of sync. The following are some such scenarios:

• After an upgrade, there are several orders/shipments already in the system which are not indexed. As these orders/shipments were created when Sterling Search Index was not active, they are not tracked anywhere, not even in YFS_Awaiting_Index. The index is out-of-sync.
• Pre-existing orders/shipments: The Sterling Search Index feature was enabled after several order/shipments were created. This can happen, for example, in the case of a fresh installation when the feature was not enabled upfront.
• The index is modified to add more indexable attributes. If the new attribute corresponds to a column that is already present in the database, and has values populated for pre-existing records, the index will now be out of sync. For example, if the ADDRESS_ID field is added to the index, and if there are existing orders with value populated in that field, then searching for orders with ADDRESS_ID in the search criteria may return only partial records.
• Sterling Search Index feature is disabled for some time. In such a case, the Sterling Search Index will not know which orders/shipments were created during the time this feature was disabled.
• Orders/shipments are migrated to the OMS system, without using OMS APIs. Since the Sterling Search Index works through OMS APIs, it will not know of such orders/shipments. The index will become out of sync.

To track ‘in sync’, we use the YFS_Index_Sync statistics table. Since order/shipment entities are enterprise specific, this table holds an ‘InSync’ flag per enterprise, per index. When the index becomes out of sync for any of the above reasons, searching from the index becomes unreliable. In such situations, the search API should bypass the search index and process all enterprise shards directly. This is achieved through the use of the InSync flag. Enterprise-specific synchronization status can be managed through the Index Management Console.

In some scenarios, InSync will be marked ‘N’ automatically by the Sterling Search Index. Such scenarios, and actions that you need to take, are described below.

• When any application/agent/integration server is started with the Sterling Search Index feature disabled (yfs.ssi.enabled=N), all enterprises will be marked out of sync for all indexes. This is because, since the Sterling Search Index feature is disabled, it is not possible to track which orders/shipments are created that need to be indexed. Once this feature is enabled, run the SSI_MASS_SYNC mode in ‘unindexed’ mode for each enterprise for each index, and then mark all enterprises to be in-sync through the Index Management Console.
• When an index definition is changed for an index, all enterprises will be marked not-in-sync for that index. Evaluate the impact of the index definition update to see which enterprises are impacted through this change. For each enterprise there is an impact on, run SSI_MASS_SYNC agent in ‘versionsync’ mode for that enterprise for that index to re-index all orders/shipments of that enterprise with the new index definition. For each enterprise that is not impacted, or for each enterprise for which the agent has been run, mark the enterprise to be in-sync using the Index Management Console.
• When any application/agent/integration server is started for the first time after an upgrade, all enterprises will be marked out of sync for all indexes. This is because, there can potentially be preexisting orders/shipments that need to be indexed. Run the SSI_MASS_SYNC agent in ‘unindexed’ mode for all enterprises (for each index) that have preexisting order/shipment records. Then use the Index Management Console to mark each enterprise to be in sync for each index.

If an enterprise needs to be marked not in sync manually, you can use ‘Mark As Not Synchronized’ in the Index Management Console. You may need to do that under exceptional circumstances, such as if an index gets corrupted and the entire index needs to be rebuilt. This will force the search APIs to skip the search index and execute on all shards of the enterprise.

Notes:

• Only the search operation is affected by the InSync flag. Index operations continue irrespective of the status of the flag.
• If InSync is automatically set to ‘N’ by the Sterling Search Index, you will be notified through an alert with either an SSI_WARN_01 or SSI_WARN_02 error code.