IBM Support

QRadar: Performance issues caused by oversubscribed hardware resources

Troubleshooting


Problem

QRadar® SIEM installed on virtual environments can experience bad performance symptoms when the physical hardware is oversubscribed, and installed along with another virtual machines sharing CPU, Memory, and Disk IO resources,

Symptom

The following are common symptoms for this issue:
  • Slow UI or CLI responses. When using QRadar, it feels slow.
  • Alerts about system load.
  • High CPU load when running all services without a specific cause.
  • A "soft lockup" message can be found in /var/log/messages. 
    kernel: BUG: soft lockup - CPU#0 stuck for 67s!

Cause

Virtual environments are meant to maximize resource usage for computing by allocating all the resources available more than once, meaning virtual machines are competing each other for resources and the hypervisor needs to dose them. This feature is called oversubscription.
QRadar provides detailed virtual appliances system requirements based on expected outcomes, oversubscribing allocated resources leads to processing delay and performance issues despite the virtual machine hardware allocation.

Environment

QRadar SIEM Virtual Appliances

Diagnosing The Problem

Administrators can run the following steps to diagnose the problem
  1. SSH to the Console as the root user and if required to the conflicting Managed Host.
  2. Run the top command to investigate CPU usage, and check the steal time value.
    image-20221027084446-3
  3. Run the iotop command to investigate the disk speed write is under x value. In another terminal or screen session, simultaneously, run the top command and verify the "waiting for I/O time" (wa) is high.
    image-20221027084507-4
Alternatively, the issue can be diagnosed from the hypervisor. The following is an example that uses vSphere®. Although the images might differ, the same steps are applicable to other hypervisors and cloud deployments. 
  1. Consult with your virtualization team, and ask whether the QRadar virtual machine is running with another virtual machines, including other QRadar appliances. Focus on CPU, Memory, and Disk IO usage.
    1. Check how much CPU is being used by going to the Host (physical hardware) in VMware vSphere Client and click VMs, then sort by CPU usage. Here you can see primary Console by using 8.5GHz and the secondary using 1.9GHz

      Check by CPU usage
       
    2.  Use the following command to test disk write speed, we can see the normal behavior under normal load and the write speed when the CPU is being oversubscribed. 
      dd if=/dev/zero of=/store/test1.img bs=1G count=1 oflag=dsync
      Under normal circumstances the write speed can go up to 250MB/s with all QRadar services running.
      Normal Write Speed
    3. Stressing the secondary CPU to the point it uses close to 20GHz we can see taking the first spot when sorting by CPU usage in vSphere.

      High CPU Usage
    4. When we run the command to test the write speed again, the write speeds reduce over 10 times with the same conditions in the primary console. 
      Reduced Write Speed

       

Resolving The Problem

If you detect there are other resource-intensive virtual machines in the same host, the following two actions can be taken:
 
  • Move them to another host letting QRadar take advantage of all granted resources, or reduce its hardware usage to prevent resource starvation among the virtual machines.
  • Move the QRadar virtual machine to another physical host with more free resources.
Results
The QRadar virtual machine is now able to consume more available hardware resources affecting positively the performance. If the performance does not improve, administrators must do the suggestions again until improvement is seen.
If the performance does not improve, contact QRadar Support for assistance
VMware, the VMware logo, VMware Cloud Foundation, VMware Cloud Foundation Service, VMware vCenter Server, and VMware vSphere are registered trademarks or trademarks of VMware, Inc. or its subsidiaries in the United States and/or other jurisdictions.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"TS010372602","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
27 October 2022

UID

ibm16620291

Page Feedback