Stores Properties

Usage

All properties can be used and set in initparm and/or sysopts. Any value set in sysopts will override value set in initparm if present.

# IO Exit parameters for Azure

file.ioexit:\
 :name=AZ1:\
 :library=/opt/cdunix/ndm/lib/libcdjnibridge.so:\
 :home.dir=/opt/cdunix/ndm/ioexit-plugins/s3:\
 :options=-Xmx640m \
 -Dstore.providerName=AZ -Dstore.tags=’key1=AnotherValue\:key2=value2’ \
 -Djava.class.path=/opt/cdunix/ndm/ioexit-plugins/s3/cd-s3-ioexit.jar com.aricent.ibm.mft.connectdirect.s3ioexit.S3IOExitFactory:
Sysopts override in CD process

sysopts=’:store.tags=’key1=AnotherValue\:key2=value2’:’
Sysopts override in CD process

sysopts=’:store.tags=’key1=AnotherValue\:key2=value2’:’

store.tags value overrides value defined in initparm.

Property names are case sensitive when used in initparm but not case sensitive when set in sysopts.

If a property value contains “:” or “=”, value must be enclosed in quotes, and the ":" must be escaped with a backslash, "\".

propertyName=’tag=abc\:error=true’

General properties

These properties are available for all stores providers. The store.providerName is the most important of them and will trigger the right provider.

Property name (, alternate name for compatibility) Description Possible values Default value Connect Direct Integrated File Agent
store.providerName Triggers the right store service

S3: Amazon S3

AZ: Azure Blob

GS: Google Storage

COS: IBM Cloud Object Storage

S3 for compatibility with previous version YES YES
store.keyStore Keystore usage

JRE_ONLY: The cacerts file will be used

SP_ONLY: Only Secure Plus keystore will be used

JRE_SP: The cacerts file and next the Secure Plus keystore

SP_JRE : The Secure Plus keystore and next the cacerts file

JRE_ONLY YES YES
store.configFromCD Integrated file agent will get the store configuration from Connect:Direct and will not use the stores.properties content

YES

NO

NO NO, Only for Integrated File Agent YES
store.contentType, s3ioexit.contentType Object Content-Type Free None YES NO
store.contentEncoding

Object Content-Encoding

For compatibility with previous version: if *.contentType contains ‘charset’, charset value will be used for Content-Encoding but only if store.contentEncoding is empty

Free None YES NO
store.dwldRange, s3ioexit.dwldRange Size of buffer to read from the provider stream >= 5MB, <= 50MB 5MB YES NO
store.objectSize, s3ioexit.objectSize Object size If CD can’t provide it. This value can be used to calculate part size on multi parts uploads.

S3: up to 5TB

AZ: up to 4.78TB

GS: up to 5TB

COS: up to 5TB

None YES NO
store.partSize, s3ioexit.partSize Override to calculated part size

S3: not less than 5MB, up to 5GB

AZ: not less than 64KB, up to 100MB

GS: not less than 5MB, up to 5TB

COS: not less than 5MB, up to 5GB

None YES NO
store.tags Additional info to store with the object as tags or metadata. S

Tags must be enclosed into quotes and separated with semicolon.

store.tags=’key=value;otherKey=abc’

None YES NO
store.maxConnections For multipart uploads, the maximum number of parallel connections the client can use. This value is not fixed or guaranteed and may vary depending on the size of the uploaded parts and available system resources. integer 30 YES NO
store.endpointUrl

Endpoint Override

This value can be overridden by any *.endpointUrl value if specified.
    YES YES
store.endpointPort

Endpoint port Override

This value can be overridden by any *.endpointPort value if specified.
    YES YES
store.endpointSecure

Endpoint will https or http

This value can be overridden by any *.endpointSecure value if specified.
YES, NO YES YES YES

Azure Blob properties (az.*)

Property name Description Possible values Default value Connect Direct Integrated File Agent
az.connectionString

The connection string includes the full set of info to connect to the service.

Value must be enclosed in quotes.

Example:

az.connectionString=’DefaultEndpointsProtocol=
https;AccountName=
cduioexit;AccountKey=abcd1r4BIZQlahie2V3cFqTg==;
BlobEndpoint=
https\://cduioexit.blob.core.windows.net/;

QueueEndpoint
=https\://cduioexit.queue.core.windows.net/;
TableEndpoint=
https\://cduioexit.table.core.windows.net/;
FileEndpoint=
https\://cduioexit.file.core.windows.net/’

None YES YES
az.applicationId Additional info application can provide Free None YES YES
az.accountName Credentials account name provided by Azure account None YES YES
az.accountKey Credentials account key provided by Azure account None YES YES
az.sasToken Credentials SAS token provided by Azure account None YES YES
az.workloadIdClientId Credentials managed Identity client ID provided by Azure account None YES YES
az.workloadIdClientId Credentials Workload Identity client ID provided by Azure account None YES, only when running on Azure YES, only when running on Azure
az.workloadTenantId Workload tenant ID provided by Azure account None YES, only when running on Azure YES, only when running on Azure
az.workloadService
TokenFilePath
File path to the service token file for workload identity provided by Azure account None YES, only when running on Azure YES, only when running on Azure
az.endpointUrl Endpoint info to override default endpoint. Mainly used when using Azurite (Use Azurite emulator for local Azure Storage development | Microsoft Docs)   None YES YES
az.endpointPort Endpoint port   None YES YES
az.endpointSecure Endpoint will use https or http YES | NO YES YES YES
az.accessTier Object storage class HOT, COOL, ARCHIVE None (inferred from bucket) YES NO

Google Storage properties (gs.*)

Property name Description Possible values Default value Connect Direct Integrated File Agent
gs.credentialsPath Path to the json credentials file Provided by Google account None YES YES
gs.projectId Additional info application can provide Free None YES YES
gs.storageClass Object storage class STANDARD, NEARLINE, COLDLINE, ARCHIVE None (inferred from bucket) YES NO
gs.partUploadFolder

GCP SDK does not provide an API for multipart uploads as other cloud providers do. Instead, CD creates parts with unique names and then composes them into the final object.

Unlike other cloud providers, these parts are not hidden, which means scanning tools may detect them as separate objects. To address this issue, a new property is now available to store these temporary parts in a dedicated folder.

When this property is enabled, parts are stored in the specified folder using the following naming pattern:container/'partUploadFolderValue'/objectKey.uniqueId.Part.n

A valid folder name None YES NO
gs.composeDelay

The Object Store Service uses the compose API provided by the Google SDK to merge uploaded parts into the final object. This API has a rate limit of one call per second for an object, and in some cases, two consecutive calls may be too fast. To address this issue, a delay and retry mechanism have been implemented when this error occurs.

Delay applied for two consecutive part uploads.

milliseconds 1000 YES NO
gs.composeRetries See gs.composeDelay

Number of retry attempts when a delay error occurs.

integer 10 YES NO
gs.endpointUrl Endpoint override     YES YES
gs.endpointPort Endpoint port override     YES YES
gs.endpointSecure Endpoint will use https or http YES, NO YES YES YES

IBM Cloud Object Storage properties (cos.*)

Property name Description Possible values Default value Connect Direct Integrated File Agent
cos.credentialsPath Path to the json credentials file   None YES YES
cos.serviceInstanceId CredentialsService Instance ID   None YES YES
cos.apiKey Credentials API key   None YES YES
cos.hmacAccessKey AWS S3 credentials hmac access key   None YES YES
cos.hmacSecretKey AWS S3 credentials hmac secret key   None YES YES
cos.profilePath AWS S3 credential file with hmac keys and profiles   None YES YES
cos.profileName AWS S3 credential file with hmac keys, profile name to use Profile names available in credentials file default YES YES
cos.endpointUrl Endpoint override   None YES YES
cos.endpointPort Endpoint port override   None YES YES
cos.endpointSecure Endpoint will use https or http YES, NO YES YES YES
cos.location Data Center location: used to dynamically build endpoint (when not overridden by cos.endpoint*) See Locations for resource deployment | IBM Cloud Docs   YES YES
cos.endpointType Endpoint Type, used to dynamically build endpoint (when not overridden by cos.endpoint*) DIRECT, PRIVATE, PUBLIC PUBLIC YES YES
cos.storageClass Object storage class Accelerated, DeepArchive, Glacier, IntelligentTiering, OneZoneInFrequentAccess, Standard, StandardInFrequentAccess Inferred from bucket YES NO
cos.sseS3 Server side encryption requested YES, NO NO YES NO
cos.virtualHostedUri

Endpoint format for bucket access, path style access (https://xxx.com/bucket-name/key-name) or virtual hosted (https://bucket-name.xxx.com/key-name)

Only works without endpoint override.

YES, NO YES YES YES

Amazon S3 properties (s3.*)

Property name Description Possible values Default value Connect Direct Integrated File Agent
s3.accessKey AWS S3 credentials hmac access key provided by Amazon account None YES YES
s3.secretKey AWS S3 credentials hmac secret key provided by Amazon account None YES YES
s3.roleArn Role arn to assume provided by Amazon account None YES YES
s3.roleProfile Role profile with credentials provided by Amazon account None YES YES
s3.roleDuration Role duration in seconds From 900 to 43200 None YES YES
s3.profilePath AWS S3 credential file with hmac keys and profiles   None YES YES
s3.configPath AWS S3 credential additional config file   None YES YES
s3.profileName AWS S3 credential file with hmac keys, profile name to use Profile names available in merged credentials file and config file default YES YES
s3.region AWS region Will be retrieve from profile If not provided None YES YES
s3.endpointUrl Endpoint override     YES YES
s3.endpointPort Endpoint port override     YES YES
s3.endpointSecure Endpoint will use https or http YES, NO YES YES YES
s3.storageClass Object storage class Deep_Archive, Glacier, Glacier_IR Intelligent_Tiering, OneZone_IA, Outposts, Reduced_Redundancy, Standard, Standard_IA Inferred from bucket YES NO
s3.sseS3 Server side encryption requested with SSE-S3 YES, NO NO YES NO
s3.virtualHostedUri

Endpoint format for bucket access, path style access (https://xxx.com/bucket-name/key-name) or virtual hosted (https://bucket-name.xxx.com/key-name)

Only works without endpoint override.

YES, NO YES YES YES
s3.useFipsEndpoint S3 FIPS endpoints must be used. See FIPS - Amazon Web Services (AWS) for more details. YES, NO NO YES YES
s3.proxyScheme

Default S3 http clients proxy scheme is HTTP. Only system properties http_proxyHost, http_proxyPort, http_proxyUser, http_proxyPassword or environment variable HTTP_PROXY can be set to establish a non-secure connection to a proxy.

For a proxy secure connection, system properties https_proxyHost, https_proxyPort, https_proxyUser, https_proxyPassword or environment variable HTTPS_PROXY, proxy scheme must be "HTTPS" but this value can't be set thru a system property.

s3.proxyScheme allows this override.

HTTP, HTTPS HTTP YES YES