Adding an Amazon CloudFront log source by using the Amazon Web Services protocol and Kinesis Data Streams

If you want to collect AWS CloudTrail logs from Amazon Kinesis Data Streams, add a log source on the QRadar® Console so that Amazon AWS CloudTrail can communicate with QRadar by using the Amazon Web Services protocol.

Procedure

  1. Use the following table describes the parameters that require specific values to collect audit events from Amazon AWS CloudTrail by using the Amazon Web Services protocol:
    Table 1. Amazon Web Services log source parameters for Amazon Kinesis Data Streams
    Parameter Description
    Protocol Configuration Select Amazon Web Services from the Protocol Configuration list.
    Authentication Method
    Access Key ID/Secret Key
    Standard authentication that can be used from anywhere.
    EC2 Instance IAM Role
    If your QRadar managed host is running in an AWS EC2 instance, choosing this option uses the IAM role from the metadata that is assigned to the instance for authentication. No keys are required. This method works only for managed hosts that are running within an AWS EC2 container.
    Access Key

    The Access Key ID that was generated when you configured the security credentials for your AWS user account.

    If you selected Access Key ID / Secret Key or Assume IAM Role, the Access Key parameter is displayed.
    Secret Key

    The Secret Key that was generated when you configured the security credentials for your AWS user account.

    If you selected Access Key ID / Secret Key or Assume IAM Role, the Secret Key parameter is displayed.
    Assume an IAM Role Enable this option to authenticate with an Access Key or EC2 instance IAM Role. Then, you can temporarily assume an IAM Role for access.
    Assume Role ARN The full ARN of the role to assume. It must begin with arn: and can't contain any leading or trailing spaces, or spaces within the ARN.

    If you enabled Assume an IAM Role, the Assume Role ARN parameter is displayed.
    Assume Role Session Name The session name of the role to assume. The default is QRadarAWSSession. Leave as the default if you don't need to change it. This parameter can contain only upper and lowercase alphanumeric characters, underscores, or any of the following characters: =,.@-

    If you enabled Assume an IAM Role, the Assume Role Session Name parameter is displayed.
    Regions Toggle each region that is associated with the Amazon Web Service that you want to collect logs from.
    AWS Service From the AWS Service list, select Kinesis Data Streams.
    Kinesis Data Stream

    The Kinesis Data Stream from which to consume data.
    Enable Kinesis Advanced Options Enable the following optional advanced configuration values. Advanced option values are only used when this option is chosen; otherwise, the default values are used.
    Initial Position in Stream
    This option controls which data to pull on a newly configured log source. Select Latest to pull the latest data that is available. Select Trim Horizon to pull the oldest data that is available.
    Kinesis Worker Thread Count
    The number of worker threads to use for Kinesis Data Stream processing. Each worker thread can process approximately 10000 - 20000 events per second depending on record size and system load. If your log source is not able to process the new data in the stream, you can increase the number of threads here to a maximum of 16. The allowed range is 1 - 16. The default value is 2.
    Checkpoint Interval
    The interval (in seconds) at which to checkpoint data sequence numbers. Each record from a shard in a Kinesis Data Stream has a sequence number. Checkpointing your position allows this shard to resume processing at the same point if processing fails or a service restarts. A more frequent interval reduces data duplication but increases Amazon Dynamo DB usage. The allowed range is 1 - 3600 seconds. The default is 10 seconds.
    Kinesis Application
    Leave this option blank to have this log source consume data from all available shards in the Kinesis Data Stream. To have multiple log sources on multiple event processors scale log consumption without loss or duplication, use a common Kinesis Application across those log sources (Example: ProdKinesisConsumers).
    Partition
    Select this option to collect data from a specific partition in the Kinesis Data Stream by specifying a partition name.
    Extract Original Event

    Forwards only the original event that was added to the Kinesis Data Stream.

    Kinesis logs wrap the events that they receive with extra metadata. Select this option if you want only the original event that was sent to AWS without the additional stream metadata through Kinesis.

    The original event is the value for the message key that is extracted from the Kinesis log. The following Kinesis logs event example shows the original event that is extracted from the Kinesis log in highlighted text:

    {"owner":"123456789012","subscriptionFilters":["allEvents"],"logEvents":[{"id":"35093963143971327215510178578576502306458824699048362100","message":"{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\"AssumedRole\",\"principalId\":\"ARO1GH58EM3ESYDW3XHP6:test_session\",\"arn\":\"arn:aws:sts::123456789012:assumed-role\/CVDevABRoleToBeAssumed\/test_visibility_session\",\"accountId\":\"123456789012\",\"accessKeyId\":\"ASIAXXXXXXXXXXXXXXXX\",\"sessionContext\":{\"sessionIssuer\":{\"type\":\"Role\",\"principalId\":\"AROAXXXXXXXXXXXXXXXXX\",\"arn\":\"arn:aws:iam::123456789012:role\/CVDevABRoleToBeAssumed\",\"accountId\":\"123456789012\",\"userName\":\"CVDevABRoleToBeAssumed\"},\"webIdFederationData\":{},\"attributes\":{\"mfaAuthenticated\":\"false\",\"creationDate\":\"2019-11-13T17:01:54Z\"}}},\"eventTime\":\"2019-11-13T17:43:18Z\",\"eventSource\":\"cloudtrail.amazonaws.com\",\"eventName\":\"DescribeTrails\",\"awsRegion\":\"ap-northeast-1\",\"sourceIPAddress\":\"192.0.2.1\",\"requestParameters\":null,\"responseElements\":null,\"requestID\":\"41e62e80-b15d-4e3f-9b7e-b309084dc092\",\"eventID\":\"904b3fda-8e48-46c0-a923-f1bb2b7a2f2a\",\"readOnly\":true,\"eventType\":\"AwsApiCall\",\"recipientAccountId\":\"123456789012\"}","timestamp":1573667733143}],"messageType":"DATA_MESSAGE","logGroup":"CloudTrail\/DefaultLogGroup","logStream":"123456789012_CloudTrail_us-east-2_2"}
    Use As A Gateway Log Source If you do not want to define a custom log source identifier for events, clear the checkbox.

    If you don't select Use As A Gateway Log Source and you don't configure the Log Source Identifier Pattern, QRadar receives events as unknown generic log sources.
    Use Predictive Parsing If you enable this parameter, an algorithm extracts log source identifier patterns from events without running the regex for every event, which increases the parsing speed.
    Tip: In rare circumstances, the algorithm can make incorrect predictions. Enable predictive parsing only for log source types that you expect to receive high event rates and require faster parsing.
    Log Source Identifier Pattern

    If you selected Use As A Gateway Log Source, you can define a custom log source identifier for events that are being processed and for log sources to be automatically discovered when applicable. If you don't configure the Log Source Identifier Pattern, QRadar receives events as unknown generic log sources.

    Use key-value pairs to define the custom Log Source Identifier. The key is the Identifier Format String, which is the resulting source or origin value. The value is the associated regex pattern that is used to evaluate the current payload. This value also supports capture groups that can be used to further customize the key.

    Define multiple key-value pairs by typing each pattern on a new line. Multiple patterns are evaluated in the order that they are listed. When a match is found, a custom Log Source Identifier is displayed.

    The following examples show multiple key-value pair functions.
    Patterns
    VPC=\sREJECT\sFAILURE
    $1=\s(REJECT)\sOK
    VPC-$1-$2=\s(ACCEPT)\s(OK)
    Events
    {LogStreamName: LogStreamTest,Timestamp: 0,Message: ACCEPT OK,IngestionTime: 0,EventId: 0}
    Resulting custom log source identifier
    VPC-ACCEPT-OK
    Use Proxy

    If QRadar accesses the Amazon Web Service by using a proxy, select this option.

    If the proxy requires authentication, configure the Proxy Server, Proxy Port, Proxy Username, and Proxy Password fields.

    If the proxy does not require authentication, configure the Proxy IP or Hostname field.
    EPS Throttle

    The upper limit for the maximum number of events per second (EPS). The default is 5000.

    If the Use As A Gateway Log Source option is selected, this value is optional.

    If the EPS Throttle parameter value is left blank, no EPS limit is imposed by QRadar.
  2. To verify that QRadar is configured correctly, review the following table to see an example of a parsed event message.

    The actual CloudTrail logs are wrapped in a Kinesis Data Streams JSON payload:

    Table 2. Kinesis Data Streams sample message supported by the Amazon AWS CloudTrail DSM
    Event name Low-level category Sample log message
    Describe Trails Read Activity Attempted 
    {"owner":"123456789012","subscriptionFilters":["allEvents"],
"logEvents":[{"id":"35101382794889527301913782399021634305485606205478862909",
"message":"{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\
"AssumedRole\",\"principalId\":\"AROA3GFMEP3ESYDW3XHP6:
cloud_visibility_session\",\"arn\":\"arn:aws:sts::123456789012:
assumed-role\/CVDevABRoleToBeAssumed\/cloud_visibility_session\",
\"accountId\":\"123456789012\",\"accessKeyId\":
\"ASIA3ABCDE3E6ZUV7IF5\",\"sessionContext\":{\"sessionIssuer\":
{\"type\":\"Role\",\"principalId\":\"AROA3GABCD3ESYDW3XHP6\",\"arn\
":\"arn:aws:iam::123456789012:role\/CVDevABRoleToBeAssumed\",
\"accountId\":\"123456789012\"\"userName\":\
"CVDevABRoleToBeAssumed\"},\"webIdFederationData\":{},\
"attributes\":{\"mfaAuthenticated\":\"false\",\"creationDate\":\
"2019-11-17T13:34:07Z\"}}},\"eventTime\":\"2019-11-17T14:10:48Z\",
\"eventSource\":\"cloudtrail.amazonaws.com\",\"eventName\":\
"DescribeTrails\",\"awsRegion\":\"ap-northeast-3\",
\"sourceIPAddress\":\"192.0.2.1\",\"requestParameters\":null,
\"responseElements\":null,\"requestID\":\
"31afb5b7-6857-467a-bce4-835ee7d02ad2\",\"eventID\":
\"26caf544-010c-423a-88a1-ca71cbd243ca\",\"readOnly\"
:true,\"eventType\":\"AwsApiCall\",\"recipientAccountId\":\
"123456789012\"}","timestamp":1574000441797}],"messageType":
"DATA_MESSAGE","logGroup":"CloudTrail\/DefaultLogGroup",
"logStream":"123456789012_CloudTrail_us-east-2"}