Amazon SQS Adapter
With the Amazon SQS or Amazon Simple Queue Service adapter, a user can access Amazon Simple Queue Services. The adapter connects to the fully managed Amazon Simple Queue Service system, using AWS keys. The SQS adapter can send and fetch messages to and from the queues to which a user has access.
This document provides an overview and the functional specifications of the Amazon SQS Adapter. The Amazon Simple Queue Service (SQS) is a distributed message-queuing service supporting programmatic sending of messages via web service applications. This message-queuing service is entirely manageable and scalable and can dissociate microservices, distributed systems, and server-less applications. The function of an Amazon SQS adapter is to access queues located at various Amazon Web Services-served geographic regions to fetch messages from standard queues and also store messages in standard queues in Amazon SQS. IAM users can access SQS messages if they have the correct AWS access key and secret key, and also the queue-access permission. The SQS Adapter supports the sending and fetching of message attributes along with the message texts. SQS adapter supports standard queues having at-least-once delivery. For more information referhttps://aws.amazon.com/sqs/.
The Amazon SQS adapter does not support the predefined-schema-import option. An AWS Identity and Access Management (IAM) user can be created in AWS for accessing the SQS services. This user represents a person or application that will interact with AWS. It is not the same as the AWS account root user. For more information, refer: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html
Adapter properties and commands
This section provides the details of the adapter properties to be set to define connections and actions, and import schemas in the new web UI of HIP. The following table lists their properties, scope, whether they are required or not, and the adapter commands they map to for use in the compiled maps and map executions. “DI” scope means the property is dynamically enumerated - the adapter is invoked during action configuration to provide the set of allowed values for that property. A detailed description of the properties is provided after the table.
| Adapter property | Scope
|
Required | Corresponding adapter command must be used directly in GET and PUT functions |
|---|---|---|---|
| accessKey= access_key | CO | Yes | -AK access_key |
| secretKey= secret_key | CO | Yes | -SK secret_key |
| region= region_name | CO | Yes | -R region_name |
| queue= queue_name | DI/IC/OC/G/P | Yes | -Q queue_name |
| createQueue= create_queue | OC/P | No | -CQ |
| attributes= attributes | IC/OC/G/P | No | -A |
| timeout= timeout | IC/G | No | -LSN wait_time_in_seconds |
| limit= limit | IC/G | No | -QTY limit |
| logging=info|errors|verbose|off | IC/OC/G/P | No | -T[E|V]?[+]? [file_name] |
| append_log=true|false | IC/OC/G/P | No | -T[E|V]?[+]? [file_name] |
| log_file_name=file_name | IC/OC/G/P | No | -T[E|V]?[+]? [file_name] |
accessKey= access_key
This property specifies the access key of AWS user for the connection. The access key is used to sign programmatic requests to AWS API calls through the adapter. The access key can be used with -ACCESSKEY or -AK command.
secretKey= secret_key
This property specifies the secret key of AWS user for the connection. Like the user name and password, both the access key ID and secret access key are mandatory to authenticate SQS queue GET or PUT request. The secret key can be passed to the adapter with -SECRETKEY or -SK command.
region= region_name
The adapter uses the region name to connect to an AWS region for the IAM user account. A region can be passed to the adapter using -REGION or -R command.
queue= queue_name
This property states the SQS queue name to or from which messages need to be accessed. During action configuration, SQS Adapter provides options to select queue name among available queues in the provided region. Also, queue name property can be passed to adapter input or output cards using -QUEUE or -Q command.
createQueue= create_queue
If specified, this property indicates the SQS adapter to create a queue with the name passed in the ‘queue_name’ property, in case it doesn’t already exist. This property is optional. When the create_queue property is not specified, adapter checks if the queue already exists; if not: it throws an error. Create queue flag can be passed to adapter input or output cards using -CREATEQUEUE or -CQ command.
attributes= attributes
Attribute property specifies if the SQS message should be accessed along with the corresponding message attributes. When the attributes flag is set to true in output card, the passed data should be in predefined JSON format, else data should be in a plain text message.
{"message": "text message", "attributes": [{"name": "attribute_name", "type= "String", "value": "attribute_value"}]
When the attributes property is present in input card, SQS adapters get the data in the above JSON format; otherwise,only text message is fetched form SQS queue. Attribute flag can be passed to adapter input or output cards using -ATTRIBUTES or -A command. If it is omitted, attributes are not used. For more information on Amazon SQS message attributes, refer:https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-attributes.html
timeout= timeout
The timeout property of SQS adapter is applicable and utilized only in HIP/ITX deployments. It specifies the time duration (in seconds) to wait for before retrieving the message from the queue. The SQS adapter would listen to the queue for specified seconds. The timeout property can be used with -LSN.
limit= limit
The limit property is applicable and utilized only in HIP/ITX deployments. It specifies the total number of messages to consume. Minimum value should be 1. The value S means all available messages. If no limit is specified, adapter fetches only one message at a time from the queue. The limit can be specified using with -QTY command.
logging=info|errors|verbose|off
This property specifies the level of logging to be used for the log (trace) file produced by the adapter. The default is off. The value info means log informational and error messages, the value error means log error messages only, and the value verbose means log debug and trace level messages along with the informational and error messages.
append_log=true|false
This property specifies the flag that indicates the action to take when the specified log file already exists. When set to true, the log messages are appended to the file. When set to false: the file is truncated, and the messages are written to the empty file. The default value is true.
log_file_name=file_name
This is the name of the log file, where the log messages are written. If not specified, the default log file name m4sqs.trc is used, and the file is stored to the directory in which the executed compiled map resides.
Examples
Examples of GET map function
In the following example, assume that the adapter being used in a GET function is defined as follows:
GET("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test A", inputdata). Since the -ATTRIBUTES/-A command is not present, the adapter only text message from the queue.
Now assume the GET map function was defined as follows: GET("SQS", " -AK AKSOMEACCESSKEY -SK SkSomeSecretKey/dBLo -R US_EAST_1 -Q queue-for-test -A", inputdata). In this case the adapter retrieves message attributes along with the text message and the output data would be like below: {"messages": "It's get example", "attributes": [{"name": "attribute1", "type"= "String", "value": "attribute example"}]}