Troubleshooting
Problem
Creating a new Amazon AWS CloudTrail log source to monitor a trail with a large amount of historical log data can result in performance and disk space issues.
Cause
Creating and deploying a new Amazon AWS CloudTral log source results in the log source attempting to retrieve all the compressed log data starting from the oldest timestamp to the most recent. This causes the log data to be retrieved and temporarily stored in the /store directory prior to QRadar attempting to extract and process this data. This can result in performance and disk space issues if a large number of historical log files are found.
Resolving The Problem
To resolve the problem, the persisted session properties for the AWS CloudTrail log source can be manually updated to set the starting point for log data retrieval.
Note: Only log files with the default CloudTrail log file name format can be collected. The filename format is <AccountID>_CloudTrail_<RegionName>_<YYYYMMDDTHHmm>Z_UniqueString.<FileNameFormat>. For example, 111122223333_CloudTrail_us-east-2_20150801T0210Z_Mu0KsOhtH1ar15ZZ.json.gz.
Procedure to set a Starting Marker.
- Disable the AWS CloudTrail Log source.
- Using SSH, log in to the QRadar Console.
- If the log source is configured for an Event Processor other than the Console, SSH from the Console to the appropriate Event Processor.
- Change Directories to /store/ec/amazonaws
cd /store/ec/amazonaws
- Using vi editor, open the AWS CloudTrail session properties file for the log source of interest.
- Modify the marker property with the name of the log file you would like to have the log source start retrieving data from.
#Amazon AWS REST API compare list
#Wed Aug 02 12:02:49 ADT 2017
marker=AWSLogs/379708147527/CloudTrail/us-east-1/2017/08/02/379708147527_CloudTrail_us-east-1_20170802T0015Z_IJjTQMuj4iA5COc1.json.gz
lastPoll=1501686169203 - Type esc :wq to save the changes.
- Enable the AWS CloudTrail log source.
Results:
You have now set the starting point for CloudTrail event data retrieval.
Related information
Amazon AWS CloudTrail Documentation
QRadar: Unable to integrate Amazon AWS logs with QRadar
Configuring QRadar to collect Amazon Web Services (AWS) S3 event data
QRadar: Troubleshooting Amazon Log Source Integrations
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt0AAA","label":"Log Source"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
28 May 2024
UID
swg22006878