Deployment

This feature is automatically enabled once all the devices in the DSNET are upgraded to the corresponding ClevOS release. All the Accessers® will support the COS S3 Select requests to enable users to filter and retrieve subset of object content using simple SQL-like queries.

 feature flag that enables support for S3 Select 
@config runtime 
@type Boolean
default value is true. Set it to false to disable COS S3 Select.
s3.enable-select-support = true

COS S3 Select Requests handled by the Accesser® can be throttled using the below advanced configuration:

Concurrency Based Throttling can be done by limiting the number of query permits for S3 Select Requests as below:

Defines a baseline number of query permits for select operations. 
If set to -1, this will default to the memory-based permit calculation.
 
There is a physical cap to number of permits a system can have, but 
it varies based the JVM's memory configuration.

@config runtime
@scope private
@type int
@range [-1, 64000] 
http.control_throttle.baseline_query_permits = 20

Defines a scaling factor for query operations to be used as a multiplier 
for the number of permits. Should be a double greater than or equal to 0. 

A negative value will have no effect, a value between 0 and 1 will reduce 
the number of permits, a value greater than one will increase the number 
of permits. If the value is too high and causes the number of permits to 
be greater than 64,000, it will be capped at 64,000 

@config runtime 
@type double 
@range [0, 64000]

service.http.control_throttle.scaling_factor_query = 1

((Accesser dsnet-core application OnHeap Memory/ service.http.memory_per_operation) * service.http.control_throttle.scaling_factor_query)

OPS Based Throttling can be done by limiting the rate of input CSV/JSON/PARQUET records processed for S3 Select Requests using the below advanced configurations:

Number of CSV input records that can be processed per second. This is a device-level throttle that is shared across
all select query requests. Values <= 0 disables throttling.
@config runtime
@scope public
@type double
select.input.csv.record-rate = 465000

Max tokens/permits that can accumulate while the device is not processing any CSV input records, acting as a buffer
against bursts. This value is only meaningful when the throttle is enabled (record-processing-rate > 0).
Values < 0 has no effect (equivalent to 0).
@config runtime
@scope public
@type long
select.input.csv.record-capacity = 465000

Number of JSON input records that can be processed per second. This is a device-level throttle that is shared across
all select query requests. Values <= 0 disables throttling.
@config runtime
@scope public
@type double
select.input.json.record-rate = 465000

Max tokens/permits that can accumulate while the device is not processing any JSON input records, acting as a buffer
against bursts. This value is only meaningful when the throttle is enabled (record-processing-rate > 0).
Values < 0 has no effect (equivalent to 0).
@config runtime
@scope public
@type long
select.input.json.record-capacity = 465000

Number of PARQUET input records that can be processed per second. This is a device-level throttle that is shared
across all select query requests. Values <= 0 disables throttling.
@config runtime
@scope public
@type double
select.input.parquet.record-rate = 465000

Max tokens/permits that can accumulate while the device is not processing any PARQUET input records, acting as a
buffer against bursts. This value is only meaningful when the throttle is enabled (record-processing-rate > 0).
Values < 0 has no effect (equivalent to 0).
@config runtime
@scope public
@type long
select.input.parquet.record-capacity = 465000

OPS Based Throttling can also be done by limiting the rate of output CSV/JSON records streamed for S3 Select Requests using the below advanced configurations:

##
# Number of CSV output records that can be processed per second. This is a device-level throttle that is shared across
# all select query requests. Values <= 0 disables throttling.
# @config runtime
# @scope public
# @type double
select.output.csv.record-rate = 465000

##
# Max tokens/permits that can accumulate while the device is not processing any CSV output records, acting as a buffer
# against bursts. This value is only meaningful when the throttle is enabled (record-processing-rate > 0).
# Values < 0 has no effect (equivalent to 0).
# @config runtime
# @scope public
# @type long
select.output.csv.record-capacity = 465000

##
# Number of JSON output records that can be processed per second. This is a device-level throttle that is shared across
# all select query requests. Values <= 0 disables throttling.
# @config runtime
# @scope public
# @type double
select.output.json.record-rate = 465000

##
# Max tokens/permits that can accumulate while the device is not processing any JSON output records, acting as a buffer
# against bursts. This value is only meaningful when the throttle is enabled (record-processing-rate > 0).
# Values < 0 has no effect (equivalent to 0).
# @config runtime
# @scope public
# @type long
select.output.json.record-capacity = 465000