Data extract configuration service

The data extract service uses configured settings to determine when to run an extract, what to extract, and where to transfer generated output files. Before you can use the IBM Sterling® Order Management System data extract service, the data extract agent for the service must be configured to work within your environments.

For the data extract service to extract the data that you need, the table configurations that indicate the data to be extracted for each database table must be defined. By default, only some database tables are enabled for the data extract service. A table configuration exists for each table that is enabled for the service.

Note: For extracting data that use data extract service, in your custom extensions you must add an index on MODIFYTS column for every table for which data extract is configured.

The data extract service reads the table configuration to obtain the configured query templates and determine the data to extract for that table. The table configurations are stored within the YFS_DATA_EXTR_CFG database table. The data extract service uses a create, retrieve, update, and delete API to manage this table.

A CocDataExtractConfig service is available in the Applications Manager for completing all types of operations on database table configurations.

To view details for this service and view the table configurations, open the Applications Manager tool, and go to Process Modeling > General > Services > Coc_Data_Extract to view the service.

When you need to add or change a configuration, make the changes within your master configuration environment and then migrate the configuration to your other environments with the Configuration Deployment tool.

Templates

When you are defining a table configuration, the query templates that define the extract process for a table configuration can resemble the following structure:

Input template:
```
<DataExtractConfig 
 Action="ADD|MODIFY|DELETE|GET|LIST" 
 DataExtractConfigKey="" 
 TableName="" TaskId="" 
 Columns="" 
 FrequencyInHours="" 
 FirstRunExtractInDays=""
/>
```
Where

DataExtractConfigKey

Unique key for table configuration.

TableName

Name of the table that includes data to be extracted.

TaskId

Identifier for a table extract.
The identifier when combined with the value of the TableName attribute form a unique table configuration record. The identifier can be a maximum of 50 characters long.

FrequencyInHours

Frequency that the extract process runs for a task.
This value is set in hours. For example, a value of 12 indicates that data is extracted for the tables that are configured in the task every 12 hours. The default setting is 24.

FirstRunExtractInDays

The range, in days, from the current time for which data is extracted when the agent runs for the first time. After the agent runs for the first time, the NextStartts is set and the successive extracts happen based on the NextStartts. Therefore, even if the FirstRunExtractInDays is modified, it is not be considered after the first run.
The default value is 30. For example, if the agent runs for the first time on 7/1/2016 with the default setting, all data that is modified since 6/1/2016 is extracted.

Columns

Comma-separated list of columns for a table that includes data that is to be extracted.
Data is extracted from the columns in the order that the columns are listed.

In addition to these attributes, you can also pass the following attributes. Pass these attributes with discretion and only if you have a full understanding of their underlying utility.

IsInactive

Flag to set a configuration active or inactive.
By default, all your configurations are active. However, if you want to deactivate a configuration, you can set IsInactive=Y. You can then reactivate it by setting IsInactive=N.

NextStartts

Custom start time for data extraction.
By default, the data extract agent automatically tracks the next start time of the data to be extracted based on earlier runs. However, in specific scenarios, you can specify a custom start time by passing the attribute NextStartts=<custom start time>. This attribute is not applicable for a LIST action.

CAUTION:
It is recommended that you do NOT use the NextStartts attribute unless there is a pressing need to use a custom start time. If not used correctly, it is possible that this action might negatively impact your data extract process.

NextRun

Custom time to trigger the next run.
By default, the data extract agent automatically tracks the time when the next trigger of the data extract runs. However, in specific scenarios, you can specify a custom next run by passing the attribute NextRun=<custom next run time>. This attribute is not applicable for a LIST action.

Note: It is recommended that you do NOT use this attribute unless there is an absolute need to use a custom next run time.

Output template for an action other than LIST:


<DataExtractConfig 
 DataExtractConfigKey="" 
 TableName="" 
 TaskId="" 
 Columns="" 
 FrequencyInHours="" 
 FirstRunExtractInDays="" 
 IsInactive=""
 NextStartts=""
 NextRun=""
/>

Output template for an action that is LIST:


<DataExtractConfigList>    
  <DataExtractConfig ...as defined above />
</DataExtractConfigList>

Configuration validations

As part of the data extract service configuration, the following validations can be completed for the configuration based on which values are set for a table configuration

Action	Validation
`ADD`	When the `Action` attribute has the value `ADD`, a check is made to determine whether the configuration exists. If the configuration exists, an error occurs. All input attributes, in the input template XML, other that the `DataExtractConfigKey` attribute, are mandatory when the attribute has the value `ADD` to complete this validation.
`MODIFY`, `DELETE`, or `GET`	When the `Action` attribute has the value `MODIFY`, `DELETE`, or `GET`, a check exists is to determine whether the configuration made. If the configuration does not exist, an error occurs. The value for the `DataExtractConfigKey` attribute or both of the `TableName` and `TaskId` attributes must be passed to complete this validation.
`LIST`	When the `Action` attribute has the value `LIST`, the input attributes for the query template are compared to all configurations. All configurations that match the input attributes are listed in the output. If no attribute is passed, all configurations are listed.
`ADD` or `MODIFY`	When the `Action` attribute has the value `MODIFY` or `ADD` and the value for the `TableName` attribute is passed, the value for the attribute is compared to the CDE_VALID_TABLE common code. If the table name matches the common code, extracting data from the table is supported. Validation is waived for extended tables. A history table is considered supported when the non-history counterpart for the table is supported. You can add to the allowlist of supported tables by getting more common code values of the above CodeType.
`ADD` or `MODIFY`	When the `Action` attribute has the value `MODIFY` or `ADD` and the value for the `Columns` attribute is passed, the value for the attribute is checked to determine whether the column exists for the table. If the column does not exist, an error occurs. Do not provide any column for any table when the data type for the column is a BLOB or CLOB data type.

Note: If the extract service runs when the data extract agent is running and already processing a data extract, any modification that you make to a configuration does not affect the ongoing data extract. The changes are applied to the next trigger of the agent.

Database tables supported for data extract

Before the 22.3 release, some transaction tables were provided by default and you could add tables for data extract by using the manageCommonCode API. From the 22.3 release, by default, every table in the Sterling™ Order Management System is a valid data extract table and you do not need to add any table manually for data extract. It is recommended that you use the data extract service to extract data only from transactional tables. It is also recommended that you use the Configuration Deployment Tool (CDT) to extract data from the configuration tables and other appropriate services to fetch data for master tables.

Migrating data extract configurations

When you need to copy your data extract configurations between environments, you can use the Configuration Deployment Tool (CDT). You can use this tool at any time to migrate configuration data for the extract configuration tables YFS_DATA_EXTR_CFG and YFS_DATA_EXTR_CLM from the master configuration environment database to a target database. However, before importing data extract configurations from one environment to another, the tables and the columns for which the data extract is defined are also present in the target environment.

For more information about migrating CDT data, see Working with the master configuration.