White Papers
Abstract
Configure Connect Direct File Agent for watching S3 buckets.
Content
File Agent can access S3 buckets using 2 mechanisms.
The credentials file
Through the credential files attached to the user running File Agent. File Agent shows running user in the log, fo example:
2021-08-25 17:16:08,007 INFO - user: xxxxxxxx, locale: en_US, java: 1.8.0_171
Credential files are located in ~/.aws.
The providers.properties file
If credentials are not located in this user home but at a different place, use the providers.properties file located in the FileAgent folder and add an aws.credentialspath entry to specify the credentials file.
If multiple profiles must be used, create more providers in the properties file.
providers.properties
# AWS S3 provider - default
cdfa.provider.1=scheme=S3://
cdfa.provider.1=provider=aws-s3
cdfa.provider.1=aws.credentialspath=/a/path/to/credentials
cdfa.provider.1=aws.profile=default
# AWS S3 provider – profile1
cdfa.provider.2=scheme=S3P1://
cdfa.provider.2=provider=aws-s3
cdfa.provider.2=aws.credentialspath=/a/path/to/credentials
cdfa.provider.2=aws.profile=profile1
Watch Directories
To list objects in a bucket linked to default profile I define a watch directory using S3://abucket/…
To list objects in a bucket linked to profile profile1 I define a watch directory using S3P1://abucket/…
To list objects in a bucket linked to profile profile1 I define a watch directory using S3P1://abucket/…
Objects will be discovered as S3://… or S3P1://… depending on the watch directory used.
Discovered Files and CD process
If a process is submitted to CD and substitution variables are used , %FA_FOUND_FILE. will be the name after scheme substitution (if any) and %FA_WATCHED_FOUND_FILE. is the discovered name.
Example:
With no scheme substitution: object found is S3P1://abucket/akey
%FA_FOUND_FILE.=S3P1://abucket/akey
%FA_WATCHED_FOUND_FILE.=S3P1://abucket/akey
%FA_FOUND_FILE.=S3P1://abucket/akey
%FA_WATCHED_FOUND_FILE.=S3P1://abucket/akey
With scheme substitution: object found is S3P1://abucket/akey
%FA_FOUND_FILE.=OS3://abucket/akey
%FA_WATCHED_FOUND_FILE.=S3P1://abucket/akey
To do a scheme substitution, add outscheme= in the provider section
cdfa.provider.2=scheme=S3P1://
cdfa.provider.2=outscheme=O3://
%FA_FOUND_FILE.=OS3://abucket/akey
%FA_WATCHED_FOUND_FILE.=S3P1://abucket/akey
To do a scheme substitution, add outscheme= in the provider section
cdfa.provider.2=scheme=S3P1://
cdfa.provider.2=outscheme=O3://
[{"Line of Business":{"code":"LOB59","label":"Sustainability Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHPZT","label":"IBM Sterling Connect:Direct File Agent"},"ARM Category":[{"code":"a8m0z000000cwUeAAI","label":"CDFA"}],"Platform":[{"code":"PF016","label":"Linux"}],"Version":"1.4.0","Type":"MASTER"}]
Was this topic helpful?
Document Information
Modified date:
20 January 2022
UID
ibm16537844