Bucket logging

Bucket logging records all access requests to a bucket and stores them as log objects in a separate log bucket. The log data can be used to monitor bucket activity, detect unauthorized access, analyze usage patterns, and maintain an audit trail of bucket changes.

Logging is configured at the bucket level and can be enabled or disabled at any time. The log bucket can collect logs from multiple source buckets. To distinguish logs from different buckets, configure a unique prefix for each source bucket so that the logs are stored in separate objects within the log bucket.Before enabling logging, ensure that the log bucket is created. The log bucket cannot be the same as the source bucket, and it cannot have logging, compression, or server-side encryption (SSE-S3 or SSE-C) enabled. It must not have RequestPayer enabled. Source and log buckets must be in the same zonegroup, but they can belong to different accounts if the log bucket policy allows it.

Note: The log bucket may have object lock enabled with a default retention period. The 16-byte unique ID in the log object name is a lexicographically ordered value made up of a 10-byte counter followed by a 6-byte random alphanumeric string. If the counter is unavailable, the ID is generated entirely from a random alphanumeric string.

The logs are stored as objects in a dedicated log bucket, which can collect events from multiple source buckets. Each source bucket should be configured with a unique prefix so that logs are written to separate objects within the log bucket.

Logging modes

Bucket logging supports two modes that differ in reliability guarantees and in the bucket operations they record.

Standard mode

In Standard mode, log records are written to the log bucket after the bucket operation completes. Failures in logging do not affect the bucket operation. Logs follow the AWS-compatible standard record format, which includes information such as request ID, client IP, HTTP status, and user agent.

Journal mode

In Journal mode, log records are written before the bucket operation completes. If logging fails, the corresponding bucket operation also fails, except for specific operations where failure is ignored. The following bucket operations are supported in Journal mode:
  • PutObject, CopyObject, CompleteMultipartUpload, PutObjectAcl, PutObjectLegalHold, PutObjectRetention, and PutObjectTagging — if logging fails, the operation fails.
  • DeleteObject, DeleteObjects, and DeleteObjectTagging — the operation continues even if logging fails.
Journal mode uses a smaller, Ceph-specific record format and is intended for journaling changes and ensuring strong consistency. Journal mode supports filtering out records based on matches of the prefixes and suffixes of the logged object keys. Regular expression matching can also be used on these to create filters. Note that it may happen that the log records were successfully written but the bucket operation failed, since the logs are written.

Reliability and flushing

Log objects do not appear in the log bucket immediately. By default, log objects are rolled every five minutes or when they reach 128 MB in size.Log objects can be flushed manually. Automatic flush occurs when logging is disabled, when the configuration changes, or when the source bucket is deleted. The time (in seconds) can be set per source bucket via a Ceph extension to the REST API , or globally via the rgw_bucket_logging_obj_roll_time configuration option. If not set, the default time is 5 minutes.

Multisite

In multisite deployments, each zone maintains temporary log objects before they are flushed to the log bucket. After flushing, log objects are then replicated across all zones similar to other objects sync in a multisite.

Policies and quotas

Only the bucket owner can enable or disable logging. To use a bucket as a log bucket, its policy must allow the s3:PutObject action for the logging.s3.amazonaws.com service principal. The policy should also specify the source bucket and account that are permitted to write logs.Bucket and user quotas apply to the log bucket:
  • In Journal mode, exceeding quota causes both logging and the bucket operation to fail.
  • In Standard mode, logging is skipped, but the bucket operation continues successfully.
For example, 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowLoggingFromSourceBucket",
      "Effect": "Allow",
      "Principal": {
        "Service": "logging.s3.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::log-bucket-name/prefix*",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "source-account-id"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::source-bucket-name"
        }
      }
    }
  ]
}

Log Objects Key Format

There are two types of log object key formats: Simple and Partitioned.

The Simple log object key follows this structure: 
<prefix><year-month-day-hour-minute-second>-<16-byte-unique-id>
For example, 
fish/2024-08-06-09-40-09-0000000002AGQ6W1
The Partitioned log object key uses a hierarchical structure that includes the bucket owner, zone group, and date components: 
<prefix><source-bucket-owner>/<zone-group>/[tenant:]<source-bucket-name>/<year>/<month>/<day>/<year-month-day-hour-minute-second>-<16-byte-unique-id>
For example, 
fish/testid/default/fish-bucket/2024/08/06/2024-08-06-10-11-18-0000000011D1FGPA

Log record formats

Bucket logging supports two formats for log records, depending on the logging mode used.
Journal format
A minimal Ceph-specific record used for journaling bucket changes.
Standard format
AWS-compatible record format with extended metadata fields such as request ID, client IP, and HTTP status.
The Journal record format uses minimum amount of data for journaling bucket changes (this is a Ceph extension):
  • bucket owner (or dash if empty)
  • bucket name (or dash if empty), in the format: [tenant:]<bucket name>
  • time in the following format: [day/month/year:hour:minute:second timezone]
  • operation in the following format: WEBSITE/REST.<HTTP method>.<resource>
  • object key (or dash if empty)
  • object size (or dash if empty
  • version id (or dash if empty)
  • eTag (or dash if empty)
For example, 
testid fish [06/Aug/2024:09:40:09 +0000] REST.PUT.OBJECT myfile - 512 4cfdfc1f58e762d3e116787cb92fac60
testid fish [06/Aug/2024:09:40:28 +0000] REST.DELETE.OBJECT myfile - - 4cfdfc1f58e762d3e116787cb92fac60
The Standard record format is based on AWS Logging Record Format.
  • bucket owner (or dash if empty)
  • bucket name (or dash if empty) in the format: [tenant:]<bucket name>
  • time in the following format: [day/month/year:hour:minute:second timezone] where “timezone” is in UTC offset
  • client IP address (or dash if empty)
  • user or account (or dash if empty)
  • request ID
  • operation in the following format: WEBSITE/REST.<HTTP method>.<resource>
  • object key (or dash if empty)
  • request URI in the following format: "<HTTP method> <URI> <HTTP version>"
  • HTTP status (or dash if zero). Note that in most cases log is written before the status is known
  • error code (or dash if empty)
  • bytes sent (or dash if zero)
  • object size (or dash if zero)
  • total time (not supported, always a dash)
  • turnaround time in milliseconds
  • referer (or dash if empty)
  • user agent (or dash if empty) inside double quotes
  • version id (or dash if empty)
  • host id taken from x-amz-id-2 (or dash if empty)
  • signature version (or dash if empty)
  • cipher suite (or dash if empty)
  • authentication type (AuthHeader for regular auth, QueryString for presigned URL or dash if unauthenticated)
  • host header (or dash if empty)
  • TLS version (or dash if empty)
  • access point ARN (not supported, always a dash)
  • ACL flag (Yes if an ACL was required for authorization, otherwise dash)
For example, 
testid fish [06/Aug/2024:09:30:25 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.15085270386962380710 REST.PUT.OBJECT myfile "PUT /fish/myfile HTTP/1.1" 200 - 512 512 - - - - - - - - - localhost - -
testid fish [06/Aug/2024:09:30:51 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.7046073853138417766 REST.GET.OBJECT myfile "GET /fish/myfile HTTP/1.1" 200 - - 512 - - - - - - - - - localhost - -
testid fish [06/Aug/2024:09:30:56 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.10723158448701085570 REST.DELETE.OBJECT myfile "DELETE /fish/myfile1 HTTP/1.1" 200 - - 512 - - - - - - - - - localhost - -
For information about enabling S2 bucket logging, see S3 enable bucket logging.