IBM Support

QRadar: How to determine average event payload and record size (in bytes) (Updated)

Troubleshooting


Problem

I am curious as to what is the average size or my events for disk space estimates. Is there a method to determine this in QRadar?

Symptom

NA

Cause

NA

Environment

QRadar 7.2 or later

Diagnosing The Problem

NA

Resolving The Problem

Yes, an administrator can log in to the appliance that is processing events and run a command that will provide a decent representation of the average event size. These values can be used to determine the average size of events, which when multiplied by the EPS rate, can give a representation of how much data is written to disk in bytes.

To determine the average event payload size and record size


This will give you the size (in bytes) of the event payloads (raw) and events records (normalized) that have been written to disk since the last ECS restart.

  1. Using SSH, log in to QRadar Console as the root user.
  2. Using SSH, log in to the appliance responsible for processing events, such as a 16xx or 18xx appliance.
  3. Based on your QRadar version, type one of the following commands:

     

    For 7.2.x or later:


    Short query:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep,type=Database writer,a1=events-2" | grep "AveragePayloadSize\|AverageRecordSize"



    Full information:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep,type=Database writer,a1=events-2"

    Example screen capture of the output:

    (Click to enlarge the image)

    To determine the elapsed time since ecs-ec was last restarted:
     ps -p $(pidof ecs-ec|sed -e 's. .,.g') -o etime=
     

    For 7.3 or later:


    Short query:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events-2" | grep "AveragePayloadSize\|AverageRecordSize"

    Full information:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events-2"

    To determine elapsed time since ecs-ep was started:
    systemctl status ecs-ep | grep Active

    - The elapsed time will be at the end of the line:

    Active: active (running) since Mon 2019-01-28 07:14:01 EST; 4h 12min ago
     

This information can then be used to determine how much space is used per day in bytes with the following formula: [eps rate] * ([AveragePayloadSize in bytes] + [AverageRecordsSize in bytes]) * 86400 

86400 represents the number of seconds per day, which is multiplied against the EPS rate and the average payload size, plus the normalized record size. To determine the average EPS rate, users can click the Dashboard tab, then select the System Monitoring dashboard item. This dashboard contains and event per second and flows per minute dashboard item. To see EPS details, click the View in Log Activity link. This will give an estimate of the data size for events per day.

NOTE: As of QRadar 7.2.7  you no longer need to figure compression as a variable as all new data in QRadar received after QRadar 7.2.7 is compressed by default.



Where do you find more information?

Internal Use Only

This technote was generated by Technote Kickstart 1.1.0.83 based on Internet Security Systems PMR 21632,024,677.
View the associated PMR's text via Wellspring at:

http://eclient.lenexa.ibm.com:9082/DocFetcher/source/PMR/21632.024.677%20O14/09/22

[{"Product":{"code":"SSBQAC","label":"IBM QRadar SIEM"},"Business Unit":{"code":"BU008","label":"Security"},"Component":"General Information","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3;7.2","Edition":""}]

Document Information

Modified date:
15 February 2019

UID

swg21685322