Source replication request log

Edit online

On IBM Cloud Object Storage, queued replications are asynchronously processed by Access Pool(s) that are capable of accessing the source bucket (i.e. pools in which the containing Container Vault is deployed to). When replication is processed, a log is generated on the source Accesser device that processed that particular replication (located in replication.log ).

Table 1. Request types
Request type Description
S3.REPLICATION.SOURCE_OBJECT_SYNC S3.REPLICATION.SOURCE_OBJECT_ATTEMPT S3.REPLICATION_RETRY.SOURCE_OBJECT_SYNC S3.REPLICATION_RETRY.SOURCE_OBJECT_ATTEMPT Syncs new object version
S3.REPLICATION.SOURCE_DELETE_SYNC S3.REPLICATION.SOURCE_DELETE_ATTEMPT S3.REPLICATION_RETRY.SOURCE_DELETE_SYNC S3.REPLICATION_RETRY.SOURCE_DELETE_ATTEMPT Syncs new delete marker
S3.REPLICATION.SOURCE_TAG_SYNC S3.REPLICATION.SOURCE_TAG_ATTEMPT S3.REPLICATION_RETRY.SOURCE_TAG_SYNC S3.REPLICATION_RETRY.SOURCE_TAG_ATTEMPT Syncs tag update on an existing version
S3.REPLICATION.SOURCE_RETENTION_SYNC S3.REPLICATION.SOURCE_RETENTION_ATTEMPT S3.REPLICATION_RETRY.SOURCE_RETENTION_SYNC S3.REPLICATION_RETRY.SOURCE_RETENTION_ATTEMPT Syncs object lock retention update on an existing version
S3.REPLICATION.SOURCE_LEGAL_HOLD_SYNC S3.REPLICATION.SOURCE_LEGAL_HOLD_ATTEMPT S3.REPLICATION_RETRY.SOURCE_LEGAL_HOLD_SYNC S3.REPLICATION_RETRY.SOURCE_LEGAL_HOLD_ATTEMPT Syncs object lock legal hold on an existing version
These request type prefixes describe the context of a given replication
  • S3.REPLICATION.*
    • Replications processed from the fast-retry queue. These replications are typically processed within minutes of making the source-side modification. If the initial replication attempt failed, retries are processed every hour up to 3 additional times before being transitioned to slow-retry queue.
  • S3.REPLICATION_RETRY.*
    • Replications processed from the slow-retry queue. These replications are retried once a day, up to 30 days, by default.
The request type suffixes describe the outcome of a given replication
  • *_SYNC
    • Replication succeeded
  • *_ATTEMPT Replication failed
Table 2. Source replication request entry fields
Field name Type Required Description
replication Object Yes Contains replication-specific info. Always present for the above request types.
replication.sync_type String Yes
Type of this replication sync.
  • content - new object version
  • tag - tag update
  • delete - new delete marker
  • retention - object lock retention update
  • legal_hold - object lock legal hold update
replication.originating_request_id String Yes Request ID of the request that originally triggered this replication. Can be traced back to the request_id on the original operation that triggered this replication (insert link to "Replication-triggering operations" article).
replication.rule_id String Yes Name of the replication rule (from the source bucket replication configuration) that triggered this replication
replication.time_in_queue Long Yes How long this replication has been queued for. Measured from when this replication was originally triggered to when this replication attempt started
replication.time_to_sync Long Yes How long this replication took to sync (or transitioned out of fast retry queue). Measured from when this replication was originally triggered to when this attempt is finished.
replication.queue_attempts Long No Number of attempts made on this replication. Present when replication attempted from fast-retry queue.
replication.failure_queue_attempts Long No Number of attempts made on this replication. Present when replication attempted from slow-retry queue.
replication.operation_throttle_delay Long Yes Total delay that was applied on this replication due to replication operation throttle
replication.byte_throttle_delay Long Yes Total delay that was applied on this replication due to replication byte transfer throttle
replication.per_account_delay Long Yes Amount of penalty applied to this replication work as a result of having too many replications triggered by the source bucket owner. The penalty is enforced in terms of how much this replication is deprioritized with respect to those that are unaffected in the queue, likely taking longer until the first replication attempt is made. However, the value does not translate to actual time duration.
replication.destination_id String Yes Destination bucket name for this replication
replication.status String Yes
Result of this replication attempt.
  • success - replication succeeded
  • no-op - nothing replicated because it was unnecessary or cannot ever succeed (e.g. version already synced, newer metadata present on destination, etc)
  • user-error - replication failed due to user configuration issue (e.g. versioning suspended on destination, write permission on destination revoked for source bucket owner, etc)
  • internal-error - replication failed due to system-level issue (e.g. bad disk in source/destination storage pool, system shutting down, etc.)
replication.message String Yes Detailed description of the replication result
replication.internal_message String Yes Internal use
replication.event_id String Yes Internal use
replication.written_to_failure_index Boolean No Set to true if this replication failed enough times and transitioned to slow-retry state (daily). Field is omitted if work is still fast-retriable (hourly).
replication.remote Object No Contains information about remote replication request. May be empty if remote request was not made (e.g. failed before).
replication.remote.last_modified_time_override Long No Timestamp corresponding to when the source version was originally created. Not present on metadata syncs.
replication.remote.tag_last_modified_time_override Long Yes Timestamp corresponding to when tag was last modified on the source version.
replication.remote.version_id_override String No ID of the version that is being replicated
replication.remote.etag_override String No ETag to be set on replica. Field is present only when ETag is being overridden.
replication.remote.destination_url String No URL that was used to make the remote replication request to the destination.
replication.remote.remote_request Object No Contains information specific to the remote replication request made out to the destination. May be empty if remote request was not made (e.g. failed before).
replication.remote.remote_request.start_time Long Yes Timestamp when the remote replication request started
replication.remote.remote_request.finish_time Long Yes Timestamp when the remote replication request finished
replication.remote.remote_request.num_attempts Integer Yes Number of remote replication request attempts. The replication agent may retry a given replication request multiple times within the context of a single top-level attempt under certain conditions (e.g. if destination returned 500, etc)
replication.remote.remote_request.request_id String No

Destination-side replication request ID. May be empty if request failed before receiving a response from destination.

This can be correlated with the request_id field on the S3.REPLICATION.TARGET* request logs on the destination for debugging purposes.
replication.remote.remote_request.status Integer No Status code received from replication request made out to destination. May be empty if request failed before receiving a response from destination.
replication.remote.remote_request.error_message String No Detailed error message from replication request made out to destination. Populated if the remote replication request failed.

Sample logs

Success

{
  "duration": "120",
  "replication": {
    "originating_request_id": "21b7d20b-199e-484b-8421-cab7e99f24ff",
    "rule_id": "1",
    "time_in_queue": "1587",
    "time_to_sync": "1707",
    "queue_attempts": "1",
    "operation_throttle_delay": "0",
    "byte_throttle_delay": "0",
    "per_account_delay": "0",
    "destination_id": "container-vault-23c9457f-bb7a-4ce6-adb2-014f7d91067b",
    "sync_type": "content",
    "status": "success",
    "message": "sync occurred successfully",
    "internal_message": "success",
    "remote": {
      "last_modified_time_override": "1766041026473",
      "tag_last_modified_time_override": "1766041026473",
      "version_id_override": "0000019b-303f-7fa9-622c-ebf86426840a",
      "destination_url": "https://10.141.25.122",
      "remote_request": {
        "start_time": "1766041028079",
        "finish_time": "1766041028180",
        "num_attempts": 1,
        "request_id": "cc08b4d6-438a-46bb-a753-0264afe2128f",
        "status": 200
      }
    },
    "event_id": "db026053-7d41-4bb7-b074-f7878ee9a823"
  },
  "selective_logging_enabled": true,
  "version_name": "0000019b-303f-7fa9-622c-ebf86426840a",
  "storage_account_id": "container-user84d26950-1ac2-4529-ab19-cda642565dae",
  "storage_location_id": "271fdc19-4ca7-7d6c-1146-4c473b22d665",
  "container_name": "container-vault-3a500bb8-2a4c-460e-a9a6-7305c397d6f6",
  "container_id": "94d73e37-7cf9-404e-983b-20808e91b968",
  "container_region": "us-south",
  "request_id": "db026053-7d41-4bb7-b074-f7878ee9a823",
  "request_type": "S3.REPLICATION.SOURCE_OBJECT_SYNC",
  "time_start": "18/Dec/2025:06:57:08 +0000",
  "timestamp_start": "1766041028064",
  "time_finish": "18/Dec/2025:06:57:08 +0000",
  "timestamp_finish": "1766041028184",
  "object_name": "e31f8174-9697-4e39-bdbe-19af83f8720f",
  "status": 200,
  "object_length": "13",
  "type": "replication",
  "format": 1
}

Failure

{
  ... (common fields omitted)
 
  "replication": {
    "originating_request_id": "bfd72065-8c4c-4871-acd0-fc176195df5f",
    "rule_id": "replication-rule-945837e8-a368-4424-9def-d8fa41cb051d",
    "time_in_queue": "3467",
    "time_to_sync": "4084",
    "queue_attempts": "4",
    "operation_throttle_delay": "0",
    "per_account_delay": "0",
    "destination_id": "container-vault-dfb460c0-4d61-4c5e-a5ba-bd926fc2ad06",
    "sync_type": "content",
    "status": "user-error",
    "message": "versioning disabled on destination bucket",
    "internal_message": "versioning disabled on destination bucket",
    "written_to_failure_index": true,
    "remote": {
      "last_modified_time_override": "1766007604072",
      "tag_last_modified_time_override": "1766007604072",
      "version_id_override": "0000019b-2e41-8368-717c-fdc98cbfeea9",
      "destination_url": "https://10.141.25.122",
      "remote_request": {
        "start_time": "1766007607877",
        "finish_time": "1766007608096",
        "num_attempts": 1,
        "request_id": "96a044e8-c144-4c02-8bee-7ca2105fd0dd",
        "status": 400,
        "error_message": "Replication failed after versioning not enabled on remote"
      }
    },
    "event_id": "2bb51704-313c-4e35-9092-21575c0cbb21"
  },
  "version_name": "0000019b-2e41-8368-717c-fdc98cbfeea9",
  "container_name": "container-vault-5dd1dde6-2b14-4281-9c63-a5f790fe6eab",
  "error": "Replication failed after versioning not enabled on remote",
  "request_id": "2bb51704-313c-4e35-9092-21575c0cbb21",
  "request_type": "S3.REPLICATION.SOURCE_OBJECT_ATTEMPT",
  "object_name": "5c62ad06-4908-491c-9740-dc66bb90325f",
  "status": 409,
  "type": "replication",
 
  ...
}