クラウドオブジェクトストレージ（COS）を利用したイベント消費

このページの内容は SaaS のみに適用されます。 IBM Cloud® Object Storageによるイベント消費は、イベント数とファイルサイズの比率を最適化した複数のイベントバッチにイベントをコンパクトにすることができます。

システムによって生成される最大ファイルサイズは、おおよそ 1.5 MB、1ファイルあたり1000イベントである。 1つのプロセスがファイルサイズ全体に寄与しない場合、システムは他のバッチデータを同じファイルに含めようとする。 1つのバッチに含まれるイベントの数が多すぎる場合、それらはイベント消費のために複数のバッチにまとめられる。

大量のイベントをいくつかのイベント・バッチに分割することで、イベント・コンシューマーがデータ処理のためにファイルをJSONにパースするのに必要なメモリは少なくて済む。 IBM Cloud Object Storageエンドポイントを持つイベント・コンシューマーは、イベント・ファイルが複数のイベント・バッチで構成され、それぞれがn改行文字で区切られていることに気づくかもしれない。

イベント処理

イベントを取得するには、以下のベストプラクティスに従ってください。

COS ファイルの消費処理パフォーマンスを向上させるために、IBM は 1 つ以上のイベント JSON を 1 つの COS ファイルに集約します。 COSファイル内のイベントJSONは、"\n"改行文字で区切られます。
JSONオブジェクト間に新しい行文字を導入することで、コンシューマーはCOSファイルを部分的に読むことができる。例えば、反復して毎回50行に到達し、JSONブロックを個別にJSONオブジェクトにパースする。あるいは、COSファイルが大きなJSONの場合、コンシューマシステムはファイル全体をパースする必要がある。これは失敗のリスクを高め、エラー時のリプレー戦略を難しくする。

次のサンプル・コード・スニペットは、COSファイルを示しています。読みやすくするため、イベントとイベントの間に改行が加えられている。

{"eventbatch":[{"channel":"Order Management","identifiers":[],"events":[{"code":"supplyChange","timestamp":"2020-01-18T06:00:00.021Z","attributes":[{"name":"tenantId","type":"String","value":"tenant1"},{"name":"productID","type":"String","value":"SKU1024"},{"name":"unitOfMeasure","type":"String","value":"EACH"},{"name":"productClass","type":"String","value":"NEW"},{"name":"shipNode","type":"String","value":"Matrix-Store-001"},{"name":"shipByDate","type":"dateTime","value":"2500-01-01T00:00:00.000Z"},{"name":"lineReference","type":"String","value":""},{"name":"reference","type":"String","value":""},{"name":"referenceType","type":"String","value":""},{"name":"segment","type":"String","value":""},{"name":"segmentType","type":"String","value":""},{"name":"quantity","type":"Number","value":"1.0"},{"name":"changedQuantity","type":"Number","value":"1.0"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","type":"dateTime","value":"IGNORE_VALUE"},{"name":"eta","type":"dateTime","value":"1900-01-01T00:00:00.000Z"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]},{"code":"supplyChange","timestamp":"2020-01-03T15:48:02Z","attributes":[{"name":"tenantId","value":"tenant1","type":"String"},{"name":"productID","value":"SKU1024","type":"String"},{"name":"unitOfMeasure","value":"EACH","type":"String"},{"name":"productClass","value":"NEW","type":"String"},{"name":"lineReference","value":" ","type":"String"},{"name":"shipNode","value":"Matrix-Store-002","type":"String"},{"name":"shipByDate","value":"2500-01-01T00:00:00Z","type":"dateTime"},{"name":"reference","value":" ","type":"String"},{"name":"referenceType","value":" ","type":"String"},{"name":"quantity","value":"200.0","type":"Number"},{"name":"segment","value":" ","type":"String"},{"name":"segmentType","value":" ","type":"String"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","value":"2020-01-03T15:47:48Z","type":"dateTime"},{"name":"eta","value":"1900-01-01T00:00:00Z","type":"dateTime"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]}]}]}

{"eventbatch":[{"channel":"Order Management","identifiers":[],"events":[{"code":"supplyChange","timestamp":"2020-01-18T06:00:00.021Z","attributes":[{"name":"tenantId","type":"String","value":"tenant1"},{"name":"productID","type":"String","value":"SKU1024"},{"name":"unitOfMeasure","type":"String","value":"EACH"},{"name":"productClass","type":"String","value":"NEW"},{"name":"shipNode","type":"String","value":"Matrix-Store-001"},{"name":"shipByDate","type":"dateTime","value":"2500-01-01T00:00:00.000Z"},{"name":"lineReference","type":"String","value":""},{"name":"reference","type":"String","value":""},{"name":"referenceType","type":"String","value":""},{"name":"segment","type":"String","value":""},{"name":"segmentType","type":"String","value":""},{"name":"quantity","type":"Number","value":"1.0"},{"name":"changedQuantity","type":"Number","value":"1.0"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","type":"dateTime","value":"IGNORE_VALUE"},{"name":"eta","type":"dateTime","value":"1900-01-01T00:00:00.000Z"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]},{"code":"supplyChange","timestamp":"2020-01-03T15:48:02Z","attributes":[{"name":"tenantId","value":"tenant1","type":"String"},{"name":"productID","value":"SKU1024","type":"String"},{"name":"unitOfMeasure","value":"EACH","type":"String"},{"name":"productClass","value":"NEW","type":"String"},{"name":"lineReference","value":" ","type":"String"},{"name":"shipNode","value":"Matrix-Store-002","type":"String"},{"name":"shipByDate","value":"2500-01-01T00:00:00Z","type":"dateTime"},{"name":"reference","value":" ","type":"String"},{"name":"referenceType","value":" ","type":"String"},{"name":"quantity","value":"200.0","type":"Number"},{"name":"segment","value":" ","type":"String"},{"name":"segmentType","value":" ","type":"String"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","value":"2020-01-03T15:47:48Z","type":"dateTime"},{"name":"eta","value":"1900-01-01T00:00:00Z","type":"dateTime"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]}]}]}

{"eventbatch":[{"channel":"Order Management","identifiers":[],"events":[{"code":"supplyChange","timestamp":"2020-01-18T06:00:00.021Z","attributes":[{"name":"tenantId","type":"String","value":"tenant1"},{"name":"productID","type":"String","value":"SKU1024"},{"name":"unitOfMeasure","type":"String","value":"EACH"},{"name":"productClass","type":"String","value":"NEW"},{"name":"shipNode","type":"String","value":"Matrix-Store-001"},{"name":"shipByDate","type":"dateTime","value":"2500-01-01T00:00:00.000Z"},{"name":"lineReference","type":"String","value":""},{"name":"reference","type":"String","value":""},{"name":"referenceType","type":"String","value":""},{"name":"segment","type":"String","value":""},{"name":"segmentType","type":"String","value":""},{"name":"quantity","type":"Number","value":"1.0"},{"name":"changedQuantity","type":"Number","value":"1.0"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","type":"dateTime","value":"IGNORE_VALUE"},{"name":"eta","type":"dateTime","value":"1900-01-01T00:00:00.000Z"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]},{"code":"supplyChange","timestamp":"2020-01-03T15:48:02Z","attributes":[{"name":"tenantId","value":"tenant1","type":"String"},{"name":"productID","value":"SKU1024","type":"String"},{"name":"unitOfMeasure","value":"EACH","type":"String"},{"name":"productClass","value":"NEW","type":"String"},{"name":"lineReference","value":" ","type":"String"},{"name":"shipNode","value":"Matrix-Store-002","type":"String"},{"name":"shipByDate","value":"2500-01-01T00:00:00Z","type":"dateTime"},{"name":"reference","value":" ","type":"String"},{"name":"referenceType","value":" ","type":"String"},{"name":"quantity","value":"200.0","type":"Number"},{"name":"segment","value":" ","type":"String"},{"name":"segmentType","value":" ","type":"String"},{"name":"type","type":"String","value":"ONHAND_BACKROOM_01"},{"name":"baseType","type":"String","value":"ONHAND"},{"name":"tagNumber","type":"String","value":"|||"},{"name":"updatedTs","value":"2020-01-03T15:47:48Z","type":"dateTime"},{"name":"eta","value":"1900-01-01T00:00:00Z","type":"dateTime"},{"name":"adjustmentReason","type":"String","value":"ADJUSTMENT"}]}]}]}

JSONオブジェクト（またはeventbatch）間の新しい行文字の導入は、コンシューマがCOSファイルを部分的に読み取ることを可能にし、処理失敗時にCOSイベントファイル全体を再生するリスクを最小限に抑える。

例えば、イベント・コンシューマーは、最初の5行またはイベント・バッチを処理し、EOFに達するまで反復することができる。エラーが発生した場合、消費者は5行のセットを再生するだけでよく、ファイル全体を再処理する必要はない。