QRadar: AWS Cloudtrail displays error "No new files matching the directory prefix and file pattern"

Resolving The Problem

• Make sure that you use the most current AWS CloudTrail DSM and Protocols.
  • For QRadar 7.4 
  • For QRadar 7.3
• Refer to the  IBM® Documentation for Amazon AWS CloudTrail log source on the QRadar Console using a directory prefix

1. Log in to the QRadar Console as admin user.
2. Click the Admin tab > Log Source Management.
3. Click +New Log Source > Single Log Source.
4. Click Amazon® AWS CloudTrail.
5. Click Select a protocol type > Amazon® AWS S3 REST API.
6. Using the IBM Documentation, follow the instructions for configuring the Log Source Parameters. 
7. Add a Log Source Identifier.
8. Configure the parameters for Authentication method, Access Key ID, and Secret Key.
9. Configure all AWS S3 Collection method parameters
10. Configure Event Format by using these examples,
    1. Event Type
       In AWS S3 buckets, you can send AWS events and other event types. The various AWS DSMs in QRadar parse only audit events. Using a custom DSM or Universal DSM, you can also pull other event types.  If you are sending audit events, make sure to select the correct event format in the log source configuration.  If the events you are trying to pull are not supported event types, for example, .txt, try the Event Generator parameter LINEBYLINE. 
       The files in the bucket are assumed to be text files with one event on each line.
    2. Unzipped events

       In order for QRadar to be able to pull the logs, the events must be zipped: .zip,.gzip,.gz. The compressed type format might vary based on the selected event format.
       • AWS CloudTrail®JSON - Files that contain JSON formatted events for Amazon® Cloud Trail® use only .json.gz files.  In the case of Universal or custom DSMs, this could include non-AWS JSON-formatted events as well.
       • LINEBYLINE -  Compression that support gzip use the extensions .gz, .gzip, or zip (.zip).
       • AWS VPC Flow Logs - Compression is used with txt.gz files only. This is used only in the case of pulling actual VPC flows.
       • AWS Network Firewall Logs - Files that contain AWS Network Firewall Alert or Flow logs. This option sends flow logs to the Network Activity tab and sends alert logs as events to the Log Activity tab in QRadar. The Amazon® AWS Network Firewall DSM parses the logs. If your system is not licensed for flows, use the Event generator parameter LINEBYLINE so that the DSM can parse the AWS Network Firewall logs.
       • W3C - Cisco® Cloud Web Services DSM uses only files with .gz extensions. 
       • Cisco® Umbrella CSV - The Cisco® Umbrella DSM use files ending only with the .gz extension.
    3. Directory Prefix
       Ensure the directory prefix field matches the directory prefix where the logs are saved in the AWS S3 Bucket.  Within that directory, the subdirectories can be only in a date format, such as \YEAR\MONTH\DAY. For example, \2021\06\20.  The directory prefix method does not traverse any other directory names.
    4. RegEx - File Pattern

       Configure a RegEx to match the log extension. It is recommended to use Regex testing tools to ensure the Regex is configured correctly. For example,

       This is the name format for the logs: thisislog1.zip
       The regex can be: .*?\.zip

       Results

       After the AWS log source configuration is corrected, events are pulled and the error message is not longer displayed.