History Mass Sync agent

Use the OSI_HISTORY_MASS_SYNC agent to migrate all the existing history orders to Order Search and stage the orders to OSI_AWAITING_ARCHIVE table for archival.

Run the OSI_HISTORY_MASS_SYNC agent immediately after enabling the built-in integration with Order Search and Order Archival.

As Order Search is the single source of truth for all the orders that are present in Sterling™ Order Management System Software and orders that are archived to Order Archival, it is important to run this agent as one time activity to migrate all the existing history orders to Order Search before archiving or making the orders eligible for archival.

For better management of the history orders in Order Search, you must consider the following available criteria:
  • SaveIndexName: You can create indexes with specific names in Order Search to migrate history orders of a particular year or for a year range. To migrate history orders, it is mandatory to provide the Save Index Name.
  • SyncFromYear and SyncToYear: An order for which the order date is between the range of year specified by using this criteria are picked up for processing. For history order migration, it is mandatory to specify the SyncFrom and SyncTo year.
Note:
  • You can run multiple instances of OSI_HISTORY_MASS_SYNC agent with different criteria parameters, but make sure that they do not define criteria with overlapping SyncFromYear and SyncToYear criteria parameters.
  • The OSI_HISTORY_MASS_SYNC agent is an enterprise agent.
  • Before running the Order Archival agent to archive orders, migrate all the existing history orders to Order Search and back up the corresponding Order Search indexes.
  • The OSI_HISTORY_MASS_SYNC agent processes the history order records, which are not already staged for archival in the OSI_AWAITNG_ARHIVE table.
  • Each executeJob runs on a ‘batch’. Each such batch corresponds to a range of entity records, for example order records from a particular colony. Furthermore, batches from different colonies are interspersed. Therefore, you can configure multiple agent threads spanning across multiple agent servers for faster indexing.
  • Since the agent runs for several records of the entity tables in each executeJob call, multiple failure scenarios can exist:
    • If critical problems are encountered that affect all index attempts, for example if the index server is unreachable, the executeJob of the agent fails with an exception.
    • If individual documents fail, failure is reported though the Exception Notification framework. Such records are reattempted in the subsequent trigger of this agent.
  • History Mass Sync agent looks at records from YFS_ORDER_HEADER_H and OSI_AWAITING_ARCHIVE tables as part of getJob to prepare the executeJob messages. It is recommended to closely monitor the performance of database queries that are executed from the History Mass Sync agent and add additional indices to get optimum throughput. Based on the testing that IBM carried with specific data setup (15 million orders in YFS_ORDER_HEADER_H table), following additional indices, helped improve the query performance.
    • An index on column ENTITY_KEY of OSI_AWAITING_ARCHIVE table
    • An index on columns ORDER_DATE and DRAFT_ORDER_FLAG of YFS_ORDER_HEADER_H table
  • Before running the agent, ensure that you set the ORDERSERVICE_USE_INDEX_NAME_IN_REQUEST property in Order Search. Once the agent run is completed and all the existing history orders are migrated, reset the property.
  • Draft orders are not supported for search and archival. Hence, this agent does not process history draft orders.

Attributes

The following table describes the History Mass Sync agent attributes:

Attribute Value
Base Transaction ID OSI_HISTORY_MASS_SYNC
Base Process Type GENERAL

Criteria Parameters

The following table describes the History Mass Sync agent criteria parameters:

Parameter Name Description
Action Required. This parameter triggers the transaction. The default value is "Get".
BatchSize Number of records to process in a single executeJob. An execute message is created for this batch of records. The default value is 500.
NumBatchesToBuffer Number of batches (execute messages) to create in one iteration of getJobs. The default value is 1000.
Number of Records To Buffer Maximum number of records to retrieve from database in one database query. The default value is 50000.
EnterpriseCode The enterprise code of the enterprise for which the History Mass Sync agent must be run.
IndexName Name of the index. For example, Order.
ColonyId Indicates the colony ID.
Collect Pending Jobs The History Mass Sync agent does not provide the pending jobs count for monitoring.
SyncFromYear A year in the yyyy format or a year and month in the yyyyMM format, from which orders are picked up for processing based on the OrderDate. If only the year is provided, the default month is January ('01').
SyncToYear A year in the yyyy format or a year and month in the yyyyMM format, up to which orders are picked up for processing based on the OrderDate. If only the year is provided, the default month is December ('12').
SaveIndexName Name of the index created in Order Search that is used by History Mass Sync agent to migrate the history orders.
Note:

History Mass Sync agent is designed to create job messages from a huge volume of data. For example, in case of orders, there can be about 10 to 20 million orders in the order history table. To create jobs, the records are fetched from an ordered list, to ensure processing of every record and to avoid duplicate attempts. Fetching smaller set of data in an ordered fashion from million records leads to numerous highly expensive database queries. Therefore, the total time that is required to create jobs for such million records is longer. However, the time required to fetch 1000 or 50000 records does not differ much and therefore you must set a higher value for the Number of Records To Buffer parameter. This helps to pull more data from the database in one attempt, which leads to lesser database queries and results in creating the required messages quickly.

The default parameter values create 1000 jobs to process 500,000 records. The number of records to process is deduced from NumBatchesToBuffer multiplied by BatchSize. However, as the default Number of Records To Buffer is 50000, it invokes only 10 database queries to create 1000 messages. You can double both NumBatchesToBuffer and Number of Records To Buffer to double the number of records to process from one iteration. However, it is recommended to double the value of Number of Records To Buffer, to get the same job done in reduced time.

Do not increase the BatchSize to a higher value as it increases the HTTP payload to Order Service.

Keep the following considerations in mind to match your expectations, when you alter the parameter values.
  • Number of Records To Buffer must be a multiple of BatchSize.
  • NumBatchesToBuffer must be a multiple of the batches created from the Number of Records To Buffer, which is Number of Records To Buffer divided by BatchSize.

Events Raised

None

Statistics Tracked

None

Pending Job Count

None