Stats files

The IBM Cloud® Object Storage Scanner tracks each LIST process status to a stats file.

During a scan, the Scanner runs multiple processes. Each LIST processes and tracks the progress, saves the next_key, and optionally the next_version to a stats file named task.stats that is stored with the log files in the /opt/ibm/metaocean/data/connections/cos/replay/output/data directory.

{
     "estimated_object_count": 1000,
     "list_objects_size": 100,
      "next_key": "",
      "next_version": "",
      "prefix": "",
      "scan_type": "Object Scan",
      "status": "Complete",
      "total_bytes_output": 1126809,
      "total_bytes_scanned": 1126809,
      "total_objects_output": 47,
      "total_objects_scanned": 47,
      "vault_name": "dsmgmt-sp1",
      "vault_uuid": "868daa21-9e56-4c41-b6fd-845a4c85cea9"
}

From the Scanner, you can start, stop, recover files from a crash, and restart at the point where the scan was interrupted.

When you start the scanner:

1. Processing of the Scanner continues from next_key and next_version.
2. Queue of the Notifier is optimized by reloading from the files in the data folder instead of requerying the dsNet.
3. Batches that were processed partially are reprocessed. Duplicate Kafka notifications might occur, but are handled safely by the IBM Spectrum® Discover system.