IBM Support

QRadar: How to pull AWS CloudTrail logs from a user specified point.

Troubleshooting


Problem

Creating a new Amazon AWS CloudTrail log source to monitor a trail with a large amount of historical log data can result in performance and disk space issues.

Cause

Creating and deploying a new Amazon AWS CloudTral log source results in the log source attempting to retrieve all the compressed log data starting from the oldest timestamp to the most recent. This causes the log data to be retrieved and temporarily stored in the /store directory prior to QRadar attempting to extract and process this data. This can result in performance and disk space issues if a large number of historical log files are found.

Resolving The Problem

To resolve the problem, the persisted session properties for AWS CloudTrail log source can be manually updated to set the starting point for log data retrieval.

Procedure to set a Starting Marker.

  1. Disable the AWS CloudTrail Log source.
  2. Using SSH log in to the QRadar Console.
  3. If the log source is configured for an Event Processor other than the Console, SSH from the Console to the appropriate Event Processor.
  4. Change Directories to /store/ec/amazonaws
    cd /store/ec/amazonaws
  5. Using vi editor open the AWS CloudTrail session properties file for the log source of interest.
  6. Modify marker property with the name of the log file you would like to have the log source start retrieving data from.
    #Amazon AWS REST API compare list
    #Wed Aug 02 12:02:49 ADT 2017
    marker=AWSLogs/379708147527/CloudTrail/us-east-1/2017/08/02/379708147527_CloudTrail_us-east-1_20170802T0015Z_IJjTQMuj4iA5COc1.json.gz
    lastPoll=1501686169203
  7. Type esc :wq to save the changes.
  8. Enable the AWS CloudTrail log source.

Results:
You have now set the starting point CloudTrail event data retrieval.


Where do you find more information?





Related information

Amazon AWS CloudTrail Documentation

QRadar: Unable to integrate Amazon AWS logs with QRadar
Configuring QRadar to collect Amazon Web Services (AWS) S3 event data
QRadar: Troubleshooting Amazon Log Source Integrations

[{"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Integrations - 3rd Party","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.2;7.3","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
16 June 2018

UID

swg22006878