IBM Support

QRadar: How to determine average event payload and record size (in bytes) (Updated)

Troubleshooting


Problem

I am curious as to what is the average size or my events for disk space estimates. Is there a method to determine this in QRadar?

Resolving The Problem

Yes, an administrator can log in to the appliance that is processing events and run a command that provides a representation of the average event size. These values can be used to determine the average size of events, which when multiplied by the EPS rate, can give a representation of how much data is written to disk in bytes.

To determine the average event payload size and record size
The procedure listed gives you the size (in bytes) of the event payloads (raw) and events records (normalized) that have been written to disk since the last ECS restart.

 

For 7.3 and later:

  1. Use SSH to log in to QRadar Console as the root user.
  2. Optional. Open an SSH session from the Console to the managed host responsible for processing events, such as a 16xx or 18xx appliance.
  3. Based on the type query you choose to run, use one of the following commands:
    Short query:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events-*.$" -a AveragePayloadSize -a AverageRecordSize 
      [root@QRadar74Console ~]# /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events-*.$' -a AveragePayloadSize -a AverageRecordSize

com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events-1
---------------------------------------------------------------------------
AverageRecordSize: 516.5108596449492
AveragePayloadSize: 1372.353294178426
    Full information:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events-*.$"
     
      [root@QRadar74Console ~]# /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events-2"
com.q1labs.ariel:application=ecs-ep.ecs-ep,type=Database writer,a1=events--2
--------------------------------------------------------------------------- 
AverageRecordSize: 214.42109268543226  
LocalNode: Local DataNode, 66593 records sent  
DatabaseRoot: /store/ariel/events/records  
AveragePayloadSize: 385.28398188922847  
ProcessedRecordCount: 66593  
ConnectedNodes:  Local DataNode, 66593 records sent    
TimeOnline: 0:10:55.748  
StoredRecordsCount: 66593  
WrittenFilesCount: 4  
PayloadRoot: /store/ariel/events/payloads  
StoredIntervalRecordsCount: 10954  
WrittenPayloadFilesCount: 4  
DroppedRecordsCount: 0  
AverageRate: 101.67 records per second  
DisconnectedNodes:    
BucketInfo:
  4. To determine the average event payload size and record size on an Event or Flow Processor
    /opt/qradar/support/jmx.sh -p 7799 -b "Database writer.*events-.$" -a AveragePayloadSize -a AverageRecordSize --noheader 
    # /opt/qradar/support/jmx.sh -p 7799 -b "Database writer.*events-.$" -a AveragePayloadSize -a 
AverageRecordSize --noheader

AveragePayloadSize: 414.2000000000001
AverageRecordSize: 254.0
  5. To determine the average event payload size and record size on a DataNode
    /opt/qradar/support/jmx.sh -p 7794 -b "Database writer.*events-.$" -a AveragePayloadSize -a AverageRecordSize --noheader 
    /opt/qradar/support/jmx.sh -p 7794 -b "Database writer.*events-.$" -a AveragePayloadSize -a AverageRecordSize --noheader

AveragePayloadSize: 413.6869262949478
AverageRecordSize: 253.88505446623094
  6. To determine Health Metrics
    1. log in to the QRadar Console.
    2. Click the Log Activity tab.
    3. Add the AQL Query to the Advanced Search. 
      SELECT Hostname, "Component Type", "Metric ID", CASE "Metric ID" WHEN 'AveragePayloadSizeEvents' 
THEN LONG(AVG(Value)) WHEN 'AverageRecordSizeEvents' THEN LONG(AVG(Value)) ELSE 0 END 
AS AverageValueInBytes FROM events WHERE devicetype = 368 AND "Metric ID" IN 
('AveragePayloadSizeEvents','AverageRecordSizeEvents') GROUP BY Hostname, 
"Component Type", "Metric ID" ORDER BY Hostname, "Metric ID" LAST 5 MINUTES
    4. Click Search.
    5. Set the Values for the graphs to AverageValueInBytes.
    6. Click Update Details.

      Results
      Graphs are created to display the Health metrics for the last five minutes.

      image 9996
       
  7. To determine elapsed time since ecs-ep was started:
      systemctl status ecs-ep | grep Active
    The elapsed time will be at the end of the line:
    Active: active (running) since Mon 2019-01-28 07:14:01 EST; 4h 12min ago

For QRadar 7.2.8

  1. Use SSH to log in to QRadar Console as the root user.
  2. Use SSH, from the Console to log in to the appliance responsible for processing events, such as a 16xx or 18xx appliance.
  3. Based on the type query you choose to run, use one of the following commands:
    Short query:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep,type=Database writer,a1=events-2" | grep "AveragePayloadSize\|AverageRecordSize"Full information:
    /opt/qradar/support/jmx.sh -p 7799 -b "com.q1labs.ariel:application=ecs-ep,type=Database writer,a1=events-2"

    Example screen capture of the output:


    To determine the elapsed time since ecs-ec was last restarted:
   ps -p $(pidof ecs-ep|sed -e 's. .,.g') -o etime=

This information can then be used to determine how much space is used per day in bytes with the following formula: [eps rate] * ([AveragePayloadSize in bytes] + [AverageRecordsSize in bytes]) * 86400 

86400 represents the number of seconds per day, which is multiplied against the EPS rate and the average payload size, plus the normalized record size. To determine the average EPS rate, users can click the Dashboard tab, then select the System Monitoring dashboard item. This dashboard contains an event per second and flows per minute dashboard item. To see EPS details, click the View in Log Activity link. This will give an estimate of the data size for events per day.

NOTE: As of QRadar V7.2.7 or later you no longer need to include compression in your calculations as all new data is compressed by default.

[{"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"General Information","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3;7.2","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
24 May 2021

UID

swg21685322

Page Feedback