How To
Summary
In a QRadar environment, it is observed that when you add new log sources, the events per second (EPS) that a
QRadar Event Processor or Event Collector need to handle increases.
To match the increased EPS, sometimes you need do increase CPU cores particularly if the Event Processor, Event Collector, or both are VM installations.
To understand the supported minimum and suggested CPU cores needed to handle a particular amount of EPS, review the Event Processor requirements section in System requirements for virtual appliances
https://www.ibm.com/docs/en/qsip/7.4?topic=installations-system-requirements-virtual-appliances
Environment
For example, let's consider you have QRadar SIEM Event Processor Virtual 16XX with 8 CPU cores, handling close to 5000 EPS to occasionally reaching 8000 EPS
Since, it is an EP, it also needs to respond to search queries, perform rule match, and so on.
Since, it is an EP, it also needs to respond to search queries, perform rule match, and so on.
In such case, when there are too many processes vying for CPU resources, you are likely to observe SAR sentinel messages such as:
Aug 4 00:43:28 ::ffff:N.N.N.N [hostcontext.hostcontext] [Thread-242] com.q1labs.hostcontext.sar.SarSentinel: [WARN] [NOT:0150124100][N.N.N.N/- -] [-/- -]System load over 1 minute has an average of 17.4 over the past 5 intervals, and has exceeded the configured threshold of 10.8. To resolve: If your system continues to exhibit this behavior, please contact Customer Support.
Aug 4 00:34:24 ::ffff:N.N.N.N [hostcontext.hostcontext] [Thread-242] com.q1labs.hostcontext.sar.SarSentinel: [WARN] [NOT:0150124100][N.N.N.N/- -] [-/- -]System load over 5 minutes has an average of 15.6 over the past 5 intervals, and has exceeded the configured threshold of 9. To resolve: If your system continues to exhibit this behavior, please contact Customer Support.
Aug 4 00:42:16 ::ffff:N.N.N.N [hostcontext.hostcontext] [Thread-242] com.q1labs.hostcontext.sar.SarSentinel: [WARN] [NOT:0150124100][N.N.N.N/- -] [-/- -]System load over 15 minutes has an average of 13.5 over the past 5 intervals, and has exceeded the configured threshold of 7.8. To resolve: If your system continues to exhibit this behavior, please contact Customer Support.
The load values are verified against the thresholds in the following file:
[root@<hostname> ~]# head -3 /opt/qradar/conf/hostcontext.sar_thresholds.conf
LOADAVERAGE1 System load over 1 minute GT 10.8 If your system continues to exhibit this behavior, please contact Customer Support. 1 1 1
LOADAVERAGE5 System load over 5 minutes GT 9 If your system continues to exhibit this behavior, please contact Customer Support. 1 2 1
LOADAVERAGE15 System load over 15 minutes GT 7.8 If your system continues to exhibit this behavior, please contact Customer Support. 1 3 1
You can verify the CPU cores on the system, by using the command:
[root@<hostname> ~]# nproc
8
You might plan to increase the CPU core to 24 (as suggested in the CPU requirement) after you review the resource shortage. More often than not after you increase the CPU to 24, you might still get the SAR sentinel warning message. The reason being, QRadar device tries to verify against the load average values in the /opt/qradar/conf/hostcontext.sar_thresholds.conf file, which is still based on previous 8 CPU cores.
To verify the latest CPU cores, you can use the nproc command.
[root@<hostname> ~]# nproc
24
You need to update the system load thresholds in the /opt/qradar/conf/hostcontext.sar_thresholds.conf file.
Steps
To update the system load values in the /opt/qradar/conf/hostcontext.sar_thresholds.conf file from 7.4.1 FixPack 1 version and later, perform the following steps:
1) If the following files exist on the system in question, move them by using the following command:
mkdir -p /store/temp_backup
mv /store/configservices/deployed/globalconfig/hostcontext.sar_thresholds.conf /store/temp_backup
mv /store/configservices/staging/globalconfig/hostcontext.sar_thresholds.conf /store/temp_backup
2) Run the following command to update system load threshold values based on the new CPU core count:
cat /opt/qradar/conf/hostcontext.sar_thresholds.conf | awk -v CPUS=$(cat /proc/cpuinfo | grep -c processor) -F '\t' 'BEGIN{OFS="\t"} $1 == "LOADAVERAGE1" {$4=CPUS*1.35 ; print} $1 == "LOADAVERAGE5" {$4=CPUS*1.125 ; print} $1 == "LOADAVERAGE15" {$4=CPUS*0.975 ; print} $1 !~ /LOADAVERAGE15$|LOADAVERAGE[15]$/ { print }' > /tmp/hostcontext.sar_thresholds.conf
3) Run the following command to apply the changes to the /opt/qradar/conf/hostcontext.sar_thresholds.conf file:
install -o nobody -g nobody -m 644 /tmp/hostcontext.sar_thresholds.conf /opt/qradar/conf/hostcontext.sar_thresholds.conf
4) Restart the hostcontext service:
systemctl restart hostcontext
5) Once, the hostcontext and all the services managed by hostcontext are in running state, confirm the new system load thresholds are showing in the hostcontext.sar_thresholds.conf file.
[root@<hostname> ~]# head -3 /opt/qradar/conf/hostcontext.sar_thresholds.conf
LOADAVERAGE1 System load over 1 minute GT 32.4 If your system continues to exhibit this behavior, please contact Customer Support. 1 1 1
LOADAVERAGE5 System load over 5 minutes GT 27 If your system continues to exhibit this behavior, please contact Customer Support. 1 2 1
LOADAVERAGE15 System load over 15 minutes GT 23.4 If your system continues to exhibit this behavior, please contact Customer Support. 1 3 1
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtcAAA","label":"Hardware"},{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"TS006022751","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.0;and future releases"}]
Was this topic helpful?
Document Information
Modified date:
02 November 2022
UID
ibm16619595