Exporting data in sequential batches

To export multiple batches of message data to an external system, you can use the Bulk Data Exports action in the External Systems application. When you use this feature, exports are processed in order based on the sequence that you specify for the publish channels.

Before you begin

Before you set up bulk data exports, complete the following tasks:
  • Configure an endpoint to the data destination.
  • Configure and enable the external system.
  • Configure the publish channels for the export.

About this task

Data is exported sequentially based on the order of the publish channels that you specify in the Bulk Data Exports window.

Data is processed using a cron task instance in the BULKEXPORT cron task. If no cron task instance exists for the external system, a cron task instance that matches the name of the external system for exporting the data is automatically generated. If the cron task instance already exists for the external system, it is activated when you click Process.

When a cron task is active, the In Process check box is selected. No other process for this external system can be started when the cron task is active. Only one export can occur at a time.

When all data in a publish channel is processed, the data is committed. If you have multiple batches in a publish channel, data is committed only after all batches are processed.

If an error occurs for any publish channel, processing is stopped, and an error message is displayed. No data for that channel is committed until the error is corrected. You can correct the error and click Process to restart the process. When you click Process, the publish channel that has the error is reprocessed, and processing continues with the next channel in the sequence. Data is not reprocessed for the publish channels that were successfully committed before the error occurred.

When the cron task is completed, it is automatically deactivated.

Table 1. Buttons in the Bulk Data Export window
Button Function
Save Saves any changes that you made in the Bulk Data Exports window.
Cancel Closes the Bulk Data Exports window but does not save your changes. To save changes, click Save before you click Cancel.
Refresh Reloads the Bulk Data Exports window and updates the fields to display the status of the data export process.
Reset Restarts the bulk data export process for all the publish channels that are included in this external system.
Process Starts the bulk data export process.

Procedure

  1. In the External Systems application, locate and open the external system that you want to use.
  2. On the System tab, in the Outbound Sequential Queue field, specify a destination for the outbound data.
  3. From the More Actions menu, select Bulk Data Exports.
  4. For the publish channels that you want to export, add rows sequentially in the order that you want to process the data and enter details for the export in the following fields:
    Option Description
    Sequence Enter a number to specify the order in which to process the publish channel data. If you add sequence numbers in increments of ten, it is easy to insert additional publish channels if the need arises at a later date. For example, if you have sequence numbers 10 and 20, and later you want to export a publish channel between those two rows, you can add a sequence number of 15.
    Publish Channel Specify the publish channel to use for the export.
    Batch Size Optional. Specify the number of records to include in each JMS message for the selected publish channel. For example, if you have 100 messages and specify a batch size of 50, the result is 2 JMS messages. If the number of records is larger than the batch size that you specify, the export data is split into a transaction that has multiple files that are stored separately. Even though the records are processed in multiple batches, only one commit occurs for the entire publish channel.
    Export Condition Optional. Specify an SQL query to select records for export. For example, specify assetnum like '11%' and siteid='BEDFORD' to export all asset records that start with 11 and are in the Bedford site.

    The query must be specified on the primary object in the publish channel object structure.
    Export Order By Optional. Specify one or more fields by which to sort records that you export. You can specify ASC for ascending order or DESC for descending order. For example, specify assetnum DESC to order the exported data by descending asset ID.

    The ORDER BY statement must apply to the primary object in the integration object.

    Error Message If an error occurs during the export process, this field displays an error message.
    Enabled Select this field in the table row to enable the publish channel.
    Processed If this publish channel was processed, this field displays a check mark.
    User Defined Indicates whether you created the publish channel or it was predefined in the application.
  5. Click Save.
  6. Click Cancel to close the window.
  7. From the More Actions menu, select Bulk Data Export.
  8. To export the data, click Process.