S3 Select Support for COS log entry fields

Edit online

Table 1. S3 Select Support log entry fields
Field name	Type	Required	Description
query_expression	String	no	Reflects the query expression provided for the SelectObjectContent request. Redacted by default due to concerns over exposing sensitive customer information in field names
bytes_processed	long	yes	The total number of uncompressed object bytes processed
bytes_returned	long	yes	The total number of bytes of records payload data returned
bytes_scanned	long	yes	The total number of object bytes scanned
query_wait	long	yes	Time elapsed between starting the query process and first byte of selected data returned to client. Similar to storage_wait entry in existing access log format for GET requests
last_write_time	long	yes	Epoch timestamp indicating when the last write occurred on the output stream. -1 if write never happened.

Table 2. Access log summary for S3 SelectObjectContent request
AWS Request name	Query Parameters	Access Log Request Type	Required Cloud IAM opcode	Access Log for Success Includes
SelectObjectContent	select	REST.POST.SELECT	object.get	select

The following code is an example of select JSON.

"select": {
    "query_expression": "SELECT count(*) FROM S3Object s",
    "bytes_processed": "178",
    "bytes_returned": "9",
    "bytes_scanned": "178",
    "query_wait": "868",
    "last_write_time":"1695060000316"
  }

The following code is an example of select JSON for a Redacted Query Expression.

 "select": {
    "bytes_processed": "178",
    "bytes_returned": "9",
    "bytes_scanned": "178",
    "query_wait": "868"
    "last_write_time":"1695060000316"
  },

Note: The select JSON object is present in both success and error paths. Request statistics may be 0 or have associated values depending on when an error was encountered in the request. Preprocessing errors such as malformed request bodies or invalid requests will have 0 while midstream errors would reflect values at time of failure.

The following code is an example of a successful select access log.

{
  "server_name": "127.0.0.1",
  "remote_address": "0.0.0.0",
  "timestamp_start": "1684271312450",
  "timestamp_finish": "1684271313363",
  "time_start": "16/May/2023:21:08:32 +0000",
  "time_finish": "16/May/2023:21:08:33 +0000",
  "request_method": "POST",
  "request_uri": "/s3/vaultName/test.txt?select&select-type=2",
  "protocol": "HTTP/1.1",
  "status": 200,
  "response_length": "364",
  "request_length": "364",
  "request_latency": "913",
  "request_id": "00000000-0000-0000-0000-000000000000",
  "request_type": "REST.POST.SELECT",
  "interface_type": "s3",
  "stat": {
    "client_wait": 25.887,
    "storage_wait": 0.129,
    "digest": 0.0,
    "commit": 0.0,
    "turn_around_time": 459.7,
    "total_transfer": 0.0,
    "pre_transfer": 16.264,
    "post_transfer": 459.7
  },
  "object_length": "178",
  "version_name": "00000188-2664-ad55-44cd-7824dad47c67",
  "version_transient": true,
  "delete_marker": false,
  "last_modified": "2023-05-16T21:08:32.213Z",
  "last_changed": "2023-05-16T21:08:32.213Z",
  "e_tag": "ba54e550edf8cd80050f092b15ce9e6c",
  "content_embedded": true,
  "segment_count": "0",
  "object_name": "test.txt",
  "proxy_enabled": false,
  "vault_name": "vaultName",
  "is_secure": false,
  "select": {
    "query_expression": "SELECT count(*) FROM S3Object s",
    "bytes_processed": "178",
    "bytes_returned": "9",
    "bytes_scanned": "178",
    "query_wait": "868",
    "last_write_time":"1695060000316"
  },
  "principals": {
    "identity": "a1181838-b486-3015-8818-74fff4545868@00000000-0000-0000-0000-000000000000"
  },
  "type": "http",
  "format": 1
}

The following code is an example of a error select access log.

{
  "server_name": "127.0.0.1",
  "remote_address": "0.0.0.0",
  "timestamp_start": "1684272369351",
  "timestamp_finish": "1684272370112",
  "time_start": "16/May/2023:21:26:09 +0000",
  "time_finish": "16/May/2023:21:26:10 +0000",
  "request_method": "POST",
  "request_uri": "/s3/vaultName/test.txt?select&select-type=2",
  "protocol": "HTTP/1.1",
  "status": 400,
  "response_length": "342",
  "request_length": "0",
  "request_latency": "754",
  "request_id": "00000000-0000-0000-0000-000000000000",
  "request_type": "REST.POST.SELECT",
  "interface_type": "s3",
  "stat": {
    "client_wait": 10.42,
    "storage_wait": 0.0,
    "digest": 0.0,
    "commit": 0.0,
    "turn_around_time": 329.216,
    "total_transfer": 0.0,
    "pre_transfer": 16.368,
    "post_transfer": 329.216
  },
  "object_length": "383",
  "version_name": "00000188-2674-cddc-d316-3eaab9a8f1c7",
  "version_transient": true,
  "delete_marker": false,
  "last_modified": "2023-05-16T21:26:09.116Z",
  "last_changed": "2023-05-16T21:26:09.116Z",
  "e_tag": "b6242abbb6d7fa0a1bc67f13d497062a",
  "content_embedded": true,
  "segment_count": "0",
  "object_name": "test.txt",
  "error_code": "JSONParsingError",
  "proxy_enabled": false,
  "vault_name": "vaultName",
  "is_secure": false,
  "select": {
    "query_expression": "SELECT * FROM S3Object",
    "bytes_processed": "0",
    "bytes_returned": "0",
    "bytes_scanned": "0",
    "query_wait": "0",
    "last_write_time":"-1"
  },
  "principals": {
    "identity": "a1181838-b486-3015-8818-74fff4545868@00000000-0000-0000-0000-000000000000"
  },
  "type": "http",
  "format": 1
}