Stores Properties
Usage
All properties can be used and set in initparm and/or sysopts. Any value set in sysopts will override value set in initparm if present.
# IO Exit parameters for Azure
file.ioexit:\
:name=AZ1:\
:library=/opt/cdunix/ndm/lib/libcdjnibridge.so:\
:home.dir=/opt/cdunix/ndm/ioexit-plugins/s3:\
:options=-Xmx640m \
-Dstore.providerName=AZ -Dstore.tags=’key1=AnotherValue\:key2=value2’ \
-Djava.class.path=/opt/cdunix/ndm/ioexit-plugins/s3/cd-s3-ioexit.jar com.aricent.ibm.mft.connectdirect.s3ioexit.S3IOExitFactory:
Sysopts override in CD process
sysopts=’:store.tags=’key1=AnotherValue\:key2=value2’:’
Sysopts override in CD process
sysopts=’:store.tags=’key1=AnotherValue\:key2=value2’:’
store.tags
value overrides value defined in
initparm.
Property names are case sensitive when used in initparm but not case sensitive when set in
sysopts
.
If a property value contains “:” or “=”, value must be enclosed in quotes, and the ":" must be escaped with a backslash, "\".
propertyName=’tag=abc\:error=true’
General properties
These properties are available for all stores providers. The store.providerName is the most important of them and will trigger the right provider.
Property name (, alternate name for compatibility) | Description | Possible values | Default value | Connect Direct | Integrated File Agent |
---|---|---|---|---|---|
store.providerName | Triggers the right store service |
S3: Amazon S3 AZ: Azure Blob GS: Google Storage COS: IBM Cloud Object Storage |
S3 for compatibility with previous version | YES | YES |
store.keyStore | Keystore usage |
JRE_ONLY: The cacerts file will be used SP_ONLY: Only Secure Plus keystore will be used JRE_SP: The cacerts file and next the Secure Plus keystore SP_JRE : The Secure Plus keystore and next the cacerts file |
JRE_ONLY | YES | YES |
store.configFromCD | Integrated file agent will get the store configuration from Connect:Direct and will not use the stores.properties content |
YES NO |
NO | NO, Only for Integrated File Agent | YES |
store.contentType, s3ioexit.contentType | Object Content-Type | Free | None | YES | NO |
store.contentEncoding |
Object Content-Encoding For compatibility with previous version: if *.contentType contains ‘charset’, charset value will be used for Content-Encoding but only if store.contentEncoding is empty |
Free | None | YES | NO |
store.dwldRange, s3ioexit.dwldRange | Size of buffer to read from the provider stream | >= 5MB, <= 50MB | 5MB | YES | NO |
store.objectSize, s3ioexit.objectSize | Object size If CD can’t provide it. This value can be used to calculate part size on multi parts uploads. |
S3: up to 5TB AZ: up to 4.78TB GS: up to 5TB COS: up to 5TB |
None | YES | NO |
store.partSize, s3ioexit.partSize | Override to calculated part size |
S3: not less than 5MB, up to 5GB AZ: not less than 64KB, up to 100MB GS: not less than 5MB, up to 5TB COS: not less than 5MB, up to 5GB |
None | YES | NO |
store.tags | Additional info to store with the object as tags or metadata. | S Tags must be enclosed into quotes and separated with semicolon. store.tags=’key=value;otherKey=abc’ |
None | YES | NO |
store.maxConnections | For multipart uploads, the maximum number of parallel connections the client can use. This value is not fixed or guaranteed and may vary depending on the size of the uploaded parts and available system resources. | integer | 30 | YES | NO |
store.endpointUrl | Endpoint Override This value can be overridden by any*.endpointUrl value if specified. |
YES | YES | ||
store.endpointPort | Endpoint port Override This value can be overridden by any*.endpointPort value if specified. |
YES | YES | ||
store.endpointSecure | Endpoint will https or http This value can be overridden by any*.endpointSecure value if specified. |
YES, NO | YES | YES | YES |
Azure Blob properties (az.*)
Property name | Description | Possible values | Default value | Connect Direct | Integrated File Agent |
---|---|---|---|---|---|
az.connectionString |
The connection string includes the full set of info to connect to the service. Value must be enclosed in quotes. |
Example: az.connectionString=’DefaultEndpointsProtocol= QueueEndpoint |
None | YES | YES |
az.applicationId | Additional info application can provide | Free | None | YES | YES |
az.accountName | Credentials account name | provided by Azure account | None | YES | YES |
az.accountKey | Credentials account key | provided by Azure account | None | YES | YES |
az.sasToken | Credentials SAS token | provided by Azure account | None | YES | YES |
az.workloadIdClientId | Credentials managed Identity client ID | provided by Azure account | None | YES | YES |
az.workloadIdClientId | Credentials Workload Identity client ID | provided by Azure account | None | YES, only when running on Azure | YES, only when running on Azure |
az.workloadTenantId | Workload tenant ID | provided by Azure account | None | YES, only when running on Azure | YES, only when running on Azure |
az.workloadService TokenFilePath |
File path to the service token file for workload identity | provided by Azure account | None | YES, only when running on Azure | YES, only when running on Azure |
az.endpointUrl | Endpoint info to override default endpoint. Mainly used when using Azurite (Use Azurite emulator for local Azure Storage development | Microsoft Docs) | None | YES | YES | |
az.endpointPort | Endpoint port | None | YES | YES | |
az.endpointSecure | Endpoint will use https or http | YES | NO | YES | YES | YES |
az.accessTier | Object storage class | HOT, COOL, ARCHIVE | None (inferred from bucket) | YES | NO |
Google Storage properties (gs.*)
Property name | Description | Possible values | Default value | Connect Direct | Integrated File Agent |
---|---|---|---|---|---|
gs.credentialsPath | Path to the json credentials file | Provided by Google account | None | YES | YES |
gs.projectId | Additional info application can provide | Free | None | YES | YES |
gs.storageClass | Object storage class | STANDARD, NEARLINE, COLDLINE, ARCHIVE | None (inferred from bucket) | YES | NO |
gs.partUploadFolder |
GCP SDK does not provide an API for multipart uploads as other cloud providers do. Instead, CD creates parts with unique names and then composes them into the final object. Unlike other cloud providers, these parts are not hidden, which means scanning tools may detect them as separate objects. To address this issue, a new property is now available to store these temporary parts in a dedicated folder. When this property is enabled, parts are stored in the specified folder using the
following naming
pattern: |
A valid folder name | None | YES | NO |
gs.composeDelay |
The Object Store Service uses the compose API provided by the Google SDK to merge uploaded parts into the final object. This API has a rate limit of one call per second for an object, and in some cases, two consecutive calls may be too fast. To address this issue, a delay and retry mechanism have been implemented when this error occurs. Delay applied for two consecutive part uploads. |
milliseconds | 1000 | YES | NO |
gs.composeRetries | See gs.composeDelay Number of retry attempts when a delay error occurs. |
integer | 10 | YES | NO |
gs.endpointUrl | Endpoint override | YES | YES | ||
gs.endpointPort | Endpoint port override | YES | YES | ||
gs.endpointSecure | Endpoint will use https or http | YES, NO | YES | YES | YES |
IBM Cloud Object Storage properties (cos.*)
Property name | Description | Possible values | Default value | Connect Direct | Integrated File Agent |
---|---|---|---|---|---|
cos.credentialsPath | Path to the json credentials file | None | YES | YES | |
cos.serviceInstanceId | CredentialsService Instance ID | None | YES | YES | |
cos.apiKey | Credentials API key | None | YES | YES | |
cos.hmacAccessKey | AWS S3 credentials hmac access key | None | YES | YES | |
cos.hmacSecretKey | AWS S3 credentials hmac secret key | None | YES | YES | |
cos.profilePath | AWS S3 credential file with hmac keys and profiles | None | YES | YES | |
cos.profileName | AWS S3 credential file with hmac keys, profile name to use | Profile names available in credentials file | default | YES | YES |
cos.endpointUrl | Endpoint override | None | YES | YES | |
cos.endpointPort | Endpoint port override | None | YES | YES | |
cos.endpointSecure | Endpoint will use https or http | YES, NO | YES | YES | YES |
cos.location | Data Center location: used to dynamically build endpoint (when not overridden by cos.endpoint*) | See Locations for resource deployment | IBM Cloud Docs | YES | YES | |
cos.endpointType | Endpoint Type, used to dynamically build endpoint (when not overridden by cos.endpoint*) | DIRECT, PRIVATE, PUBLIC | PUBLIC | YES | YES |
cos.storageClass | Object storage class | Accelerated, DeepArchive, Glacier, IntelligentTiering, OneZoneInFrequentAccess, Standard, StandardInFrequentAccess | Inferred from bucket | YES | NO |
cos.sseS3 | Server side encryption requested | YES, NO | NO | YES | NO |
cos.virtualHostedUri |
Endpoint format for bucket access, path style access (https://xxx.com/bucket-name/key-name) or virtual hosted (https://bucket-name.xxx.com/key-name) Only works without endpoint override. |
YES, NO | YES | YES | YES |
Amazon S3 properties (s3.*)
Property name | Description | Possible values | Default value | Connect Direct | Integrated File Agent |
---|---|---|---|---|---|
s3.accessKey | AWS S3 credentials hmac access key | provided by Amazon account | None | YES | YES |
s3.secretKey | AWS S3 credentials hmac secret key | provided by Amazon account | None | YES | YES |
s3.roleArn | Role arn to assume | provided by Amazon account | None | YES | YES |
s3.roleProfile | Role profile with credentials | provided by Amazon account | None | YES | YES |
s3.roleDuration | Role duration in seconds | From 900 to 43200 | None | YES | YES |
s3.profilePath | AWS S3 credential file with hmac keys and profiles | None | YES | YES | |
s3.configPath | AWS S3 credential additional config file | None | YES | YES | |
s3.profileName | AWS S3 credential file with hmac keys, profile name to use | Profile names available in merged credentials file and config file | default | YES | YES |
s3.region | AWS region | Will be retrieve from profile If not provided | None | YES | YES |
s3.endpointUrl | Endpoint override | YES | YES | ||
s3.endpointPort | Endpoint port override | YES | YES | ||
s3.endpointSecure | Endpoint will use https or http | YES, NO | YES | YES | YES |
s3.storageClass | Object storage class | Deep_Archive, Glacier, Glacier_IR Intelligent_Tiering, OneZone_IA, Outposts, Reduced_Redundancy, Standard, Standard_IA | Inferred from bucket | YES | NO |
s3.sseS3 | Server side encryption requested with SSE-S3 | YES, NO | NO | YES | NO |
s3.virtualHostedUri |
Endpoint format for bucket access, path style access (https://xxx.com/bucket-name/key-name) or virtual hosted (https://bucket-name.xxx.com/key-name) Only works without endpoint override. |
YES, NO | YES | YES | YES |
s3.useFipsEndpoint | S3 FIPS endpoints must be used. See FIPS - Amazon Web Services (AWS) for more details. | YES, NO | NO | YES | YES |
s3.proxyScheme |
Default S3 http clients proxy scheme is HTTP. Only system properties http_proxyHost, http_proxyPort, http_proxyUser, http_proxyPassword or environment variable HTTP_PROXY can be set to establish a non-secure connection to a proxy. For a proxy secure connection, system properties https_proxyHost, https_proxyPort, https_proxyUser, https_proxyPassword or environment variable HTTPS_PROXY, proxy scheme must be "HTTPS" but this value can't be set thru a system property. s3.proxyScheme allows this override. |
HTTP, HTTPS | HTTP | YES | YES |