IBM Support

QRadar: What is the Amazon REST API

How To


Summary

The Amazon Simple Queue Service (SQS) provides a generic web service API that you can access by using any language that the AWS SDK supports. The Simple Storage Service (S3 Bucket) stores data as objects within resources called buckets. Administrators requiring events from Amazon Simple Queue Service or Amazon Simple Storage Service can now use the Amazon REST API to collect these types of events. This article outlines and clarifies common error messages and required fields.

Steps

Amazon Simple Queue Service (SQS)

The AmazonSQS builds an AmazonSQS object used to query messages. The queue has a maximum of 10 messages at a time.

Debug Enabled
Debug is enabled Status Expected response in log
Debug is enabled Success
Received [quantity of] messages from [sqs Queue URL]
Failure
No messages available from from [sqs Queue URL]
When each message is processed one by one
MessageID: [ID] Body [message body]


In each message SQS expects to receive "Record" elements from the JSON response, which could be nested in the Message element or in the Record element 

Description Expected response in debug log
If Records are not found No "Records" or "Message" element in notification - skipping
In each non-empty record, SQS looks for "s3" element. If "S3" elements are not found No "s3" element in record [message ID] - skipping
If one record has "S3" elements, SQS captures the "objectKey" and "objectSize" from "object" element, and captures the "bucketName" and "arn" from "bucket" element BucketName [bucketName] ::  BucketArn [arn] :: ObjectKey [objectKey] :: ObjectSize [objectSize]
If the objectSize is 0 or it's CloudTrail-Digest data, the entry is skipped
Object [objectKey] is [0] bytes - skipping processing
Found "/CloudTrail-Digest/" while processing AWS_CLOUD_TRAIL_JSON
 - skipping processing
If the "objectKey" value doesn't match the file name regex entered SQS continues to skip processing.
Object key [objectKey] does not match File Pattern
 [filePattern] skipping processing
SQS starts downloading matched files unless it receives a download exception in error log:
Failed to download object [objectKey] from bucket [bucketName]
 - Consecutive Failed Downloads [consecutiveFailedDownloadsNumber]
After the file is downloaded, SQS starts processing the file. See response in /var/log/qradar.log:
Processing S3 File: [file name]
SQS processes the file to its type, posts the events, and deletes the downloaded file.
qradar.log: Processed [number of] records from S3 File [file name]
qradar.java.debug: Deleting local file [file name]

Results
The processed messages are removed at this step and SQS retrieves another batch of messages until there are no messages to be polled.

Amazon Simple Storage Service (S3 Bucket)

S3 Bucket is an option that needs to be selected it is obtaining a list of the names in the bucket after the prefix and the first marker.

Description Expected response in log
S3 Bucket is an option that needs to be selected. It obtains a list of the names in the bucket after the prefix and the first marker.
Listing Objects :: Bucket [bucket name] :: Prefix [prefix] 
:: Marker [marker]
For each object returned, S3 buckets only take objects matching the file name regex and with nonzero sizes. Debug log:
Key [file path/name] does not match pattern 
[file path/name pattern] - skipping

Key [file path/name] is [0] bytes - skipping

Found [number] S3 objects from bucket [buchet name] 
with prefix [prefix] and marker [marker]
After the name list is obtained, the protocol removes processed files from the list if they are Amazon CloudTrail logs. Check debug:
Duplicate File Cache contains [duplicate file cache size]
 elements

File [file name] is in the duplicateFileCache - 
removing from list

Removed [number] files from list matching 
duplicateFileCache
After the files are processed, the protocol sets the marker to the last file name in the list or empty when list is empty. if marker is not present yet. Check qradar.log:
Initialized marker for new configuration of bucket 
[bucket name] prefix [prefix] - starting with new data after 
[file name]

Initial listing of bucket [bucket name] prefix [prefix] 
performed but no files found
If the marker is already present, the protocol continues downloading and processing files. See debug:
Attempting to download resource [file path/name]

Currently [number] threads still running

Current download stats for this interval.... 
Attempted: [attempt number], Failed: [failure number]

About to process file: [file name]
Administrators should be able to see events posted if any is in the debug:
POSTED: " [event]

At the end of processing each file, the local file gets removed and the marker gets updated. If the file is Cloud Trail, its name is added to the duplicated list too. This allows it to relist all files in the last hour to resolve major delays for Cloud Trail logs and any duplicates are removed during processing.

Description Expected response in log
When setting markers, the protocol modifies the marker to one hour before if it is Cloud Trail only. For example,
Sample file name:
AWSLogs/769160150729/CloudTrail/us-east-1/2018/06/26/769160150729_
CloudTrail_us-east-1_20180626T1230Z_r88ATd2WrNlcNvpg.json.gz
Marker:
AWSLogs/769160150729/CloudTrail/us-east-1/2018/06/26/769160150729_
CloudTrail_us-east-1_20180626T1130Z

please note 1230Z versus 1130Z in the file name part.

Amazon Rest API troubleshooting

These steps are for common Amazon Rest API problems.

Problem Solution
The duplicate cache could not be booted up or was corrupted. Remove the cache and so it can rebuild.
Suspicion of missing logs The protocol is posting what it gets back from Amazon API response. One can manually modify the marker to poll historical logs again to confirm
Potential Out of Memory As every cloud trail preserves a duplicate cache, it consumes a large amount of memory. Too many log sources residing on one Managed Host might result in Out of Memory. It can be resolved by reducing the number of log sources per Managed Host or increase ingress memory if possible.

Document Location

Worldwide

   
   

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt0AAA","label":"Log Source"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
09 November 2022

UID

ibm16838227