Introduction to Archive Service capabilities

As the e-commerce industry and online shopping is growing at a faster rate, enterprises today are facing significant challenges with exponential data growth. The avalanche of data in the database leads to an increased infrastructure cost to maintain more high speed read and write disk space. At the same time, the size of the database can affect the overall responsiveness and system performance.

Sterling Order Management System Software supports purge operation for various transactional data. You can perform the data purge operation any time based on your business requirements. Even if you purge frequently or keep lesser retention days, the data moves to the history table that shares the same table space as the main table. Basically, a frequent purge makes transactions in main tables faster, but from the database perspective, precisely the disk space, the amount of data and disk space requirement remains the same. Hence, costs are not reduced.

You can review the situations and scenarios in which the order archival capabilities might meet your business needs. Sterling Order Management System Software provides the capability to archive historical order data and its related entities through Archive Service component. The Archive Service component is built on Cassandra database, which offers low-cost storage for orders that are no longer actively used or older than the specific date for long time retention. It allows easy access to those historical order data and reduces the primary storage consumption which is typically expensive.

The order volume in Sterling Order Management System Software keeps growing day by day and the infrastructure cost incurred on the database management becomes expensive. It also impacts the data retrieval operation as system may require scanning through the very large old data.

Currently, in Sterling Order Management System Software you can either run history order purge agents to permanently remove historical orders or you can keep them in history tables for longer duration. In order to maintain the size of data low in Sterling Order Management System Software database and to make the system work efficiently, currently business provides one of the following options:
  • Implement YFSPurgeOrderExternallyUE, which is invoked as part of the Order Purge agent. This helps moving the transactional data to external storage directly without storing them in the history tables. This requires a lot of implementation effort including user exit implementation and writing custom logic to store and manage the data to external storage.
  • Write custom implementation to read data from history tables and move them to external storage. Once the data is moved, run the history purge agent to remove the data from Sterling Order Management System Software history tables

Sterling Order Management System Software provides built-in integration to archive the historical order data from the Sterling Order Management System Software database to Archive Service. This integration also takes care of removing the historical order data from the Sterling Order Management System Software database after successful archival. You can easily access the historical data stored in Archive Service with the help of APIs provided by Archive Service.

When the integration with Order Service is enabled, the Order Purge agent takes care of staging the order records in OSI_AWAITING_ARCHIVE table. After purging the order data to history tables, a record is inserted in the OSI_AWAITING_ARCHIVE table with the entity key and the date information after which the order is available for archival. The OSI_AWAITING_ARCHIVE table acts as a staging location for order archival. While inserting the record to OSI_AWAITING_ARCHIVE table, the Order Purge agent also calculates ARCHIVE_ON_DATE on which the order is available for archival. The ARCHIVE_ON_DATE attribute is derived based on the purge criteria defined for history orders. The ARCHIVE_ON_DATE attribute is calculated by adding retention days configured for history orders to purged date. If additional purge criteria is defined, then additional retention days are considered in ARCHIVE_ON_DATE calculation.

Sterling Order Management System Software provides the OSI_ORDER_ARCHIVE time-triggered transaction for order archival. You can configure this transaction in GENERAL repository. This transaction picks up the staged orders from the OSI_AWAITING_ARCHIVE table for archival based on the ARCHIVE_ON_DATE attribute. Before processing any records, the OSI_ORDER_ARCHIVE agent performs few validations to minimize any failures during archival.

After fetching records from the staging location, the order archival process fetches the corresponding order and related entities data from history tables, prepares JSON input for archival, and makes a synchronous call to Archive Service. While preparing each JSON request, the payload size and number of records are considered, which you can configure for each entity, which needs to be archived separately. The order archival process ensures that each batch of request to Archive Service does not exceed the configured size limits. You must ensure that the data size that you configure should be able to accommodate at least one record for that entity.

The order archival process archives orders in multiple logical grouping of entities. It first archives order lines and related entities. As described earlier, data is sliced into multiple requests based on batch size limits. After archiving order line, order audit is archived. If you do not want to archive your order audit data due to large size or any other considerations, you may choose to skip the archival of order audit by configuring a property. Additionally, if you want to archive any other direct subordinate entity of an order entity, you can do that by a set of defined properties. If you choose to archive such entities, they are archived after order audit archival or order line archival in case you chose to skip order audit archival.

Once order line and other entities, which are configured for separate archival are archived, the order archival process archives the order entity at last. If the archival operation is successful for all the entities for that order, the corresponding order details are removed from the history tables.

Sterling Order Management System Software also provides user exits that allow you to provide archival data for order and order line related tables for which the relationship is not defined in Sterling Order Management System Software. Once the data is deleted from Sterling Order Management System Software history tables, there is one more user exit which is invoked to notify the successful archival of your custom data. You can use these set of user exits to archive and delete your custom table data.

The order archival process also logs a set of statistics to apprise you of the failures and successful archival that happened over a period of time. Additionally, it notifies you about any exceptions or failures occurred during order archival process through alerts and events.

Note: Archival of draft orders and quotes are not supported.