Event consumption using cloud object storage (COS)

This page content applies to SaaS only. The event consumption with IBM Cloud® Object Storage allows the events to be compacted into multiple event batches that are optimized between event count to file size ratio.

The maximum file size that is produced by the system is roughly 1.5 MB or 1000 events per file. If a single process does not contribute to the full file size, the system attempts to include other batch data into the same file. If one batch contains too many events, they are batched into multiple batches for event consumption.

By breaking down large number of events into several event batches, the event consumer require lesser memory to parse file into JSONs for data processing. The event consumers with an IBM Cloud Object Storage endpoint may notice that events file consists of multiple event batch and each are separated by a \n new line character.

Event Processing

To retrieve events, follow these best practices.
  • To improve the consumption processing performance of a COS file, IBM aggregates one or more event JSON into a single COS file. The event JSON within the COS file is separated by a "\n" new line character.
  • The introduction of the new line character between JSON objects enables the consumer to partially read the COS file. For example, iterate and reach 50 lines each time and parse JSON blocks into JSON object individually. Alternatively, if the COS file is a large JSON, the consumer system needs to parse the entire file. This has an increase risk of failure and more difficult for replay strategy when there is an error.
The following sample code snippet illustrates the COS file. For readability, a new line is added between events.
{"eventbatch":[{"channel":"Order Management","identifiers":[],"events":[{"code":"supplyChange","timestamp":"2020-01-18T06:00:00.021Z","attributes":[{"name":"tenantId","type":"String","value":"tenant1"},{"name":"productID","type":"String","value":"SKU1024"},{"name":"unitOfMeasure","type":"String","value":"EACH"},{"name":"productClass","type":"String","value":"NEW"},{"name":"shipNode","type":"String","value":"Matrix-Store-001"},{"name":"shipByDate","type":"dateTime","value":"2500-01-01T00:00:00.000Z"},{"name":"lineReference","type":"String","value":""},{"name":"reference","type":"String","value":""},{"name":"referenceType","type":"String","value":""},{"name":"segment","type":"String","value":""},{"name":"segmentType","type":"String","value":""},{"name":"quantity","type":"Number","value":"1.0"},{"name":"changedQuantity","type":"Number","value":"1.0"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","type":"dateTime","value":"IGNORE_VALUE"},{"name":"eta","type":"dateTime","value":"1900-01-01T00:00:00.000Z"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]},{"code":"supplyChange","timestamp":"2020-01-03T15:48:02Z","attributes":[{"name":"tenantId","value":"tenant1","type":"String"},{"name":"productID","value":"SKU1024","type":"String"},{"name":"unitOfMeasure","value":"EACH","type":"String"},{"name":"productClass","value":"NEW","type":"String"},{"name":"lineReference","value":" ","type":"String"},{"name":"shipNode","value":"Matrix-Store-002","type":"String"},{"name":"shipByDate","value":"2500-01-01T00:00:00Z","type":"dateTime"},{"name":"reference","value":" ","type":"String"},{"name":"referenceType","value":" ","type":"String"},{"name":"quantity","value":"200.0","type":"Number"},{"name":"segment","value":" ","type":"String"},{"name":"segmentType","value":" ","type":"String"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","value":"2020-01-03T15:47:48Z","type":"dateTime"},{"name":"eta","value":"1900-01-01T00:00:00Z","type":"dateTime"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]}]}]}

{"eventbatch":[{"channel":"Order Management","identifiers":[],"events":[{"code":"supplyChange","timestamp":"2020-01-18T06:00:00.021Z","attributes":[{"name":"tenantId","type":"String","value":"tenant1"},{"name":"productID","type":"String","value":"SKU1024"},{"name":"unitOfMeasure","type":"String","value":"EACH"},{"name":"productClass","type":"String","value":"NEW"},{"name":"shipNode","type":"String","value":"Matrix-Store-001"},{"name":"shipByDate","type":"dateTime","value":"2500-01-01T00:00:00.000Z"},{"name":"lineReference","type":"String","value":""},{"name":"reference","type":"String","value":""},{"name":"referenceType","type":"String","value":""},{"name":"segment","type":"String","value":""},{"name":"segmentType","type":"String","value":""},{"name":"quantity","type":"Number","value":"1.0"},{"name":"changedQuantity","type":"Number","value":"1.0"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","type":"dateTime","value":"IGNORE_VALUE"},{"name":"eta","type":"dateTime","value":"1900-01-01T00:00:00.000Z"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]},{"code":"supplyChange","timestamp":"2020-01-03T15:48:02Z","attributes":[{"name":"tenantId","value":"tenant1","type":"String"},{"name":"productID","value":"SKU1024","type":"String"},{"name":"unitOfMeasure","value":"EACH","type":"String"},{"name":"productClass","value":"NEW","type":"String"},{"name":"lineReference","value":" ","type":"String"},{"name":"shipNode","value":"Matrix-Store-002","type":"String"},{"name":"shipByDate","value":"2500-01-01T00:00:00Z","type":"dateTime"},{"name":"reference","value":" ","type":"String"},{"name":"referenceType","value":" ","type":"String"},{"name":"quantity","value":"200.0","type":"Number"},{"name":"segment","value":" ","type":"String"},{"name":"segmentType","value":" ","type":"String"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","value":"2020-01-03T15:47:48Z","type":"dateTime"},{"name":"eta","value":"1900-01-01T00:00:00Z","type":"dateTime"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]}]}]}

{"eventbatch":[{"channel":"Order Management","identifiers":[],"events":[{"code":"supplyChange","timestamp":"2020-01-18T06:00:00.021Z","attributes":[{"name":"tenantId","type":"String","value":"tenant1"},{"name":"productID","type":"String","value":"SKU1024"},{"name":"unitOfMeasure","type":"String","value":"EACH"},{"name":"productClass","type":"String","value":"NEW"},{"name":"shipNode","type":"String","value":"Matrix-Store-001"},{"name":"shipByDate","type":"dateTime","value":"2500-01-01T00:00:00.000Z"},{"name":"lineReference","type":"String","value":""},{"name":"reference","type":"String","value":""},{"name":"referenceType","type":"String","value":""},{"name":"segment","type":"String","value":""},{"name":"segmentType","type":"String","value":""},{"name":"quantity","type":"Number","value":"1.0"},{"name":"changedQuantity","type":"Number","value":"1.0"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","type":"dateTime","value":"IGNORE_VALUE"},{"name":"eta","type":"dateTime","value":"1900-01-01T00:00:00.000Z"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]},{"code":"supplyChange","timestamp":"2020-01-03T15:48:02Z","attributes":[{"name":"tenantId","value":"tenant1","type":"String"},{"name":"productID","value":"SKU1024","type":"String"},{"name":"unitOfMeasure","value":"EACH","type":"String"},{"name":"productClass","value":"NEW","type":"String"},{"name":"lineReference","value":" ","type":"String"},{"name":"shipNode","value":"Matrix-Store-002","type":"String"},{"name":"shipByDate","value":"2500-01-01T00:00:00Z","type":"dateTime"},{"name":"reference","value":" ","type":"String"},{"name":"referenceType","value":" ","type":"String"},{"name":"quantity","value":"200.0","type":"Number"},{"name":"segment","value":" ","type":"String"},{"name":"segmentType","value":" ","type":"String"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","value":"2020-01-03T15:47:48Z","type":"dateTime"},{"name":"eta","value":"1900-01-01T00:00:00Z","type":"dateTime"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]}]}]}
The introduction of the new line character between JSON objects (or eventbatch) enables the consumer to partially read the COS file and minimize risk of replaying the entire COS event file during processing failure.

For example, the event consumer may process the first 5 lines or event batch, and iterate until the EOF is reached. When an error occurs, the consumer is only required to replay those sets of 5 lines and eliminate the need to reprocess the entire file.