Amazon VPC Flow Logs

The IBM® QRadar® integration for Amazon VPC (Virtual Private Cloud) Flow Logs collects VPC flow logs from an Amazon S3 bucket by using an SQS queue.

Important: This integration supports the default format for Amazon VPC Flow Logs and any custom formats that contain version 3, 4, or 5 fields. However, all version 2 fields must be included in your custom format. The default format includes the following fields.
${version} ${account-id} ${interface-id} ${srcaddr} ${dstadir} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}

For more information, see the Amazon VPC Flow Logs documentation.

To integrate Amazon VPC Flow Logs with QRadar, complete the following steps:
  1. If automatic updates are not enabled, RPMs are available for download from the IBM support website (http://www.ibm.com/support). Download and install the following RPMs on your QRadar Console.
    • Protocol Common RPM
    • AWS S3 REST API PROTOCOL RPM
    Important: If you are installing the RPM to enable more AWS-related VPC flow fields in the QRadar Network Activity Flow Details window, then the following services must be restarted before they are visible. You don't need to restart the services for the protocol to function.
    Hostcontext
    To restart host context, see QRadar: Hostcontext service and the impact of a service restart (https://www.ibm.com/support/pages/qradar-hostcontext-service-and-impact-service-restart).
    Tomcat
    On the Console, click the Admin tab, and then click Advanced > restart Web Server.
  2. Configure your Amazon VPC Flow Logs to publish the flow logs to an S3 bucket.
  3. Create the SQS queue that is used to receive ObjectCreated notifications from the S3 bucket that you used in step 2.
  4. Create security credentials for your AWS user account.
  5. Add an Amazon VPC Flow Logs log source on the QRadar Console.
    Important: A Flow Processor must be available and have a FPM license to receive the flow logs. VPC Flow Log does not use an EPS license. Unlike other log sources, AWS VPC Flow Log events are not sent to the Log Activity tab. They are sent to the Network Activity tab.
    Important: When the VPC Flow Logs log source is configured by using Universal DSM, it does not generate any event. In this case, the Last Event time status remains blank.

    The following table describes the parameters that require specific values to collect events from Amazon VPC Flow Logs:

    Table 1. Amazon VPC Flow Logs log source parameters
    Parameter Value
    Log Source type A custom log source type.
    Protocol Configuration Amazon AWS S3 REST API
    Target Event Collector The Event Collector or Event Processor that receives and parses the events from this log source.
    Tip: This integration collects raw event logs of Amazon VPC Flow Logs from the target AWS S3 bucket. Then, it generates IPFIX flow records and forwards the records to the VPC Flow Destination Hostname. You can use a Flow Collector or a Flow Processor as the target event collector only when it is a combined Flow Collector and Flow Processor or an all-in-one console.
    Log Source Identifier

    Type a unique name for the log source.

    The Log Source Identifier can be any valid value and does not need to reference a specific server. The Log Source Identifier can be the same value as the Log Source Name. If you configured more than one Amazon VPC flow Logs log source, you might want to name in an identifiable way. For example, you can identify the first log source as vpcflowlogs1 and the second log source as vpcflowlogs2.
    Authentication Method
    Access Key ID / Secret Key
    Standard authentication that can be used from anywhere.
    For more information about configuring security credentials, see Configuring security credentials for your AWS user account.
    EC2 Instance IAM Role
    If your managed host is running on an AWS EC2 instance, choosing this option uses the IAM Role from the instance metadata that is assigned to the instance for authentication. No keys are needed. This method works only for managed hosts that are running within an AWS EC2 container.
    Assume IAM Role Enable this option by authenticating with an Access Key or EC2 instance IAM Role. Then, you can temporarily assume an IAM Role for access. This option is available only when you use the SQS Event Notifications collection method.

    For more information about creating IAM users and assigning roles, see Creating an Identity and Access Management (IAM) user in the AWS Management Console.
    Event Format AWS VPC Flow Logs
    S3 Collection Method SQS Event Notifications
    VPC Flow Destination Hostname The hostname or IP address of the Flow Processor where you want to send the VPC logs.
    Tip: For QRadar to accept IPFIX flow traffic, you must configure a NetFlow/IPFIX flow source that uses UDP. Most deployments can use a default_Netflow flow source and set the VPC Flow Destination Hostname to the hostname of that managed host.

    If the managed host that is configured with the NetFlow/IPFIX flow source is the same as the Target Event Collector that was chosen earlier in the configuration, you can set the VPC Flow Destination Hostname to localhost.

    For more information about creating flow sources, see the IBM QRadar Administration Guide.
    VPC Flow Destination Port The port for the Flow Processor where you want to send the VPC logs.
    Important: This port must be the same as the monitoring port that is specified in the NetFlow flow source. The port for the default_Netflow flow source is 2055.
    SQS Queue URL The full URL that begins with https://, for the SQS Queue that is set up to receive notifications for ObjectCreated events from S3.
    Region Name The region that is associated with the SQS queue and S3 bucket.

    Example: us-east-1, eu-west-1, ap-northeast-3
    Show Advanced Options The default is No. Select Yes if you want to customize the event data.
    File Pattern

    This option is available when you set Show Advanced Options to Yes.

    Type a regex for the file pattern that matches the files that you want to pull; for example, .*?\.json\.gz
    Local Directory

    This option is available when you set Show Advanced Options to Yes.

    The local directory on the Target Event Collector. The directory must exist before the AWS S3 REST API PROTOCOL attempts to retrieve events.
    S3 Endpoint URL

    This option is available when you set Show Advanced Options to Yes.

    The endpoint URL that is used to query the AWS REST API.

    If your endpoint URL is different from the default, type your endpoint URL. The default is http://s3.amazonaws.com.
    Use Proxy

    If QRadar accesses the Amazon Web Service by using a proxy, enable Use Proxy.

    If the proxy requires authentication, configure the Proxy Server, Proxy Port, Proxy Username, and Proxy Password fields.

    If the proxy does not require authentication, configure the Proxy Server and Proxy Port fields.
    Recurrence How often the Amazon AWS S3 REST API Protocol connects to the Amazon cloud API, checks for new files, and if they exist, retrieves them. Every access to an AWS S3 bucket incurs a cost to the account that owns the bucket. Therefore, a smaller recurrence value increases the cost.

    Type a time interval to determine how frequently the remote directory is scanned for new event log files. The minimum value is 1 minute. The time interval can include values in hours (H), minutes (M), or days (D). For example, 2H = 2 hours, 15 M = 15 minutes.
    EPS Throttle The maximum number of events per second that are sent to the flow pipeline. The default is 5000.

    Ensure that the EPS Throttle value is higher than the incoming rate or data processing might fall behind.
  6. To send VPC flow logs to the IBM QRadar Cloud Visibility app for visualization, complete the following steps:
    1. On the Console, click the Admin tab, and then click System Configuration > System Settings.
    2. Click the QFlow Settings menu, and in the IPFix additional field encoding field, choose either the TLV or TLV and Payload format.
    3. Click Save.
    4. From the menu bar on the Admin tab, click Deploy Full Configuration and confirm your changes.
      Warning: When you deploy the full configuration, QRadar services are restarted. During this time, events and flows are not collected, and offenses are not generated.
    5. Refresh your browser.

For more information about configuring the Amazon AWS S3 REST API protocol, see Amazon AWS S3 REST API protocol configuration options.