Configuring the data extract

Before you use the IBM Sterling® Order Management System data extract service, ensure that the data extract agent for the service is configured to work in your environments.

The data extract agent is provided as a time-triggered multi-threaded agent. You can view and configure the agent within the Applications Manager.

After you configure the agent, use the Self Service to start and stop the agent. When the agent starts, the extract process triggers and runs at scheduled intervals. You can also run the extract service between the scheduled times when you want to re-create an output file or if a file is corrupted.

Procedure

Create data extract-related table configurations. For more information, see Data extract configuration service.
Configure the data extract agent settings.
1. Open Applications Manager.
2. Go to Platform > Process Modeling > General > Coc Data Extract Agent.
3. Within the Time-triggered tab, set the following JMS-related attributes.
  - Trigger the interval to 10 minutes or more.
  - Number of threads to run in parallel based on how many concurrent stable connections your file service can handle. Suggested 2 to 5 number of threads.
To update these details, set the following properties in the System Administration Console.
1. If your file storage service is an SFTP server, set up your own SFTP server such that the data extract service can transfer output files to your SFTP server. To configure this connection, update the details and location of your SFTP server in your Sterling™ Order Management System production environment.
  - yfs.yfs.coc.dataextract.sftp.hostname=hostname
  - yfs.yfs.coc.dataextract.sftp.port=port
  - yfs.yfs.coc.dataextract.sftp.dir=directory_path
  - yfs.yfs.coc.dataextract.sftp.username=user
  - yfs.yfs.coc.dataextract.sftp.password=password
  Here,
  - hostname refers to the hostname of your SFTP server.
  - port refers to the port number of your SFTP server.
  - directory_path refers to the path on your SFTP server where the output files are to be transferred.
  - user refers to the username that must be specified to connect to the SFTP server.
  - password refers to the password for the SFTP user.
2. You can use a cloud-based file storage service to extract data by using the data extract tool. Any service that is fully compliant with the Amazon AWS S3 protocol can be used. The application certifies cloud services such as Amazon AWS S3 and IBM Cloud® Object Storage for use with the tool. For other fully S3-compliant services, you can perform their certification to enable compatibility.
  To use a S3-based service, add the following configurations to set it up to receive data extract files:
  - yfs.yfs.coc.dataextract.export.mode=S3 – Specify S3 to use S3 library compatible file service. The default value is SFTP.
  - yfs.yfs.coc.dataextract.s3.bucketname – Name of the S3 bucket to which the files are extracted.
  - yfs.yfs.coc.dataextract.s3.accesskey – Access key to access the S3 bucket.
  - yfs.yfs.coc.dataextract.s3.secretkey – Secret key to access the S3 bucket.
  - yfs.yfs.coc.dataextract.s3.endpointurl – Endpoint URL to access the S3 bucket.
  - yfs.yfs.coc.dataextract.s3.location – Location to access the S3 bucket. This is optional, if the location can be resolved from the endpoint URL itself.
  - yfs.yfs.coc.dataextract.s3.region – Region to access the S3 bucket. Specify either the endpoint URL or Region.
  - yfs.yfs.coc.dataextract.s3.prefix – Optional. Custom prefix to be appended to file names that are exported to the S3 bucket.
  - yfs.yfs.coc.dataextract.s3.requesttimeout – Timeout for S3 request in milliseconds. This is optional as the default 5000 is used.
  - yfs.yfs.coc.dataextract.s3.useS3SignerType – Specify this property to true to enable data extract to Google Cloud Storage (GCS).
3. Configure the data extract tool with the following properties to store extracted files in Microsoft Azure Blob Storage. This configuration requires providing details about your Azure Blob Storage account, container, and secure access credentials. The configuration ensures that the files are exported to the correct container with appropriate security by using SAS tokens.
  - yfs.yfs.coc.dataextract.export.mode – Specify the export mode for data extraction. Set this to AzureBSRest to use Azure Blob Storage via REST API.
  - yfs.yfs.coc.dataextract.azurebs.accountname – Specify the name of your Azure Blob Storage account. This identifies the storage account where the container resides. Example value: myaccount.
  - yfs.yfs.coc.dataextract.azurebs.containername – Specify the name of the Azure Blob Storage container where files will be extracted. Example value: abc.
  - yfs.yfs.coc.dataextract.azurebs.prefix – Specify an optional custom prefix to append to file names exported to the Azure Blob Storage container. Example value: ext.
  - yfs.yfs.coc.dataextract.azurebs.sastoken – Set the SAS token to provide secure delegated access to the container.
  - Review the following guidelines for SAS tokens.
    - Generate SAS tokens at the container level.
    - Ensure permissions include racwld.
    - Use a long enough expiry for Data Extract Agent operations. Example value: sp=racwld&st=....
Optional: You can configure the data extract service for your business by setting the following optional properties in the System Administration Console.
- yfs.yfs.coc.dataextract.csv.escapedoublequote – Option to escape a double quotation mark in the extracted data with a custom string. By default, if any double quotation mark (") exists in the extracted data, it is replaced with the string, two double quotation marks (""). Using this property, you can specify a custom string to replace the double quotation marks in your data.
- yfs.yfs.coc.dataextract.fetch.maxrows – Maximum limit of records that are extracted per query in case the limit is calculated from memory is higher. By default, the value is set to 50000.
- yfs.yfs.coc.dataextract.fetch.maxminsold – Time interval in minutes before the current time for which the existing data is not extracted. The default value is 60 minutes.
  Note: If you set a value that is lower than the default value, you might lose data updates when you extract data. The lower the value, the greater the risk of data loss.
- yfs.yfs.coc.dataextract.fetch.maxhoursold – Time interval in hours, before current time, for which the existing data is not extracted. By default, the value is set to 1. This does not apply if yfs.yfs.coc.dataextract.fetch.maxminsold is set.
- yfs.yfs.coc.dataextract.csv.delimiter – Column delimiter in the exported CSV file. The default value is a comma (,)
- yfs.yfs.coc.dataextract.csv.newlinelastrecord – Add a line character after the last record in the exported CSV file. By default, the value is set to N.
- yfs.yfs.coc.dataextract.csv.firstrowheader – Add a header row that contains the names of the columns that are extracted in the exported CSV file. By default, the value is set to N.
- yfs.yfs.coc.dataextract.export.removefailedfiles – Optionally, you can remotely clean up stale extract files that are part of failed extracts by setting the value to Y. By default, the stale extract files that are part of failed extracts are not cleaned up.
- yfs.yfs.coc.dataextract.fetch.maxdaysextractatonce – retrieve long-span extracts in smaller chunks, which are defined by a specified number of days, instead of fetching data from the tracked start time to the current time in one go.
  By default, the data extract agent retrieves data that spans for a long period in a single operation. That is, from the tracked start time to the current time, particularly during first-time extracts. You can specify this property to a specific number of days to perform the long span extracts in chunks of the specified number of days. A duration of 30 days is recommended. For example, if your FirstRunExtractInDays is 365 (1 year), when you run the data extract, the complete data of 365 days is extracted in one trigger, resulting in the creation of lots of jobs. Also, even if one job fails, data extract is recommenced from the start time. However, if this property is set to 30, the data extract is done over 365/30=~13 triggers. This minimizes the chances of having to re-extract data from the beginning.
- yfs.yfs.coc.dataextract.export.minspacerequired – Configure the minimum space that is needed in the file storage service to run the data extract agent. The default value is 1024 MB. Maintain sufficient space in the file storage service, based on your business needs so that the data extract agent runs successfully. If the space is less than the configured threshold, data extract is paused. Currently, applicable only for the SFTP server.
Start the CocDataExtractServer agent.
When the agent is started, the extract process is triggered and runs at the scheduled intervals.