IBM Support

QRadar: Large numbers of assets can cause the Arc_builder to go out-of-memory on the managed host (APAR IJ00838)

Troubleshooting


Problem

This technical note provides further information for administrators on how to identify and get QRadar Support involved in cases related to APAR IJ00838: ARC_BUILDER GOES OUT OF MEMORY GOES WHEN THE ASSET CEILING NUMBER IS SET TO 5 MILLION ASSETS .

 

Symptom

Administrators who experience this issue will notice /var/log/qradar.log messages being generated on the managed host stating that the arc_builder is OutOfMemory or that the arc_builder local daemon cannot load. QRadar systems with QRadar Risk Manager enabled can experience issues on either Console or managed hosts where software updates that increase the overall asset ceiling (asset maximum) can cause the arc_builder component to go out-of-memory (OOM) and generate the exception described in APAR IJ00838.

 

Cause

QRadar appliances need to have their arc_builder memory increased using a .override file to prevent the out-of-memory condition for the arc_builder component and to prevent error messages in QRadar logs.

 

Environment

  • QRadar appliances at V7.2.8
  • QRadar appliances at V7.3.0 Patch 4 (7.3.1. 20180507202600) and above.


 

Diagnosing The Problem

To diagnose this issue, administrators can review the QRadar logs to determine if they are seeing arc_builder or OutOfMemoryError messages in the logs on their QRadar appliances.

For example:
zless /var/log/qradar.old/qradar.log.2.gz| grep arc_builder
or
tail -f /var/log/qradar.error | grep OutOfMemoryError
 
IMPORTANT: Administrators should never extract log files or error files found in /var/log/qradar.old on the local appliance, instead use the zless command. Extracting logs to the local disk will cause issues with logrotate and can run the partition out of space on the local host, stopping services and interrupt event/flow questions. 
 
Example error message from /var/log/qradar.log: 
hostname arc_builder[22051]: Caused by:
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
hostname arc_builder[22051]: at com.q1labs.semsources.filters.arc.ArcBuilder.init(ArcBuilder.java:240)
hostname arc_builder[22051]: ... 5 more
hostname arc_builder[22051]: Caused by: java.lang.OutOfMemoryError: Java heap space
hostname arc_builder[22051]: at gnu.trove.TLongHashSet.rehash(TLongHashSet.java:169)
hostname arc_builder[22051]: at gnu.trove.THash.postInsertHook(THash.java:359)
hostname arc_builder[22051]: at gnu.trove.TLongHashSet.add(TLongHashSet.java:154)
hostname arc_builder[22051]: at com.q1labs.semsources.filters.arc.NetworkModelsServices.loadExistingPortData
(NetworkModelsServices.java:405)
hostname arc_builder[22051]: at com.q1labs.semsources.filters.arc.NetworkModelsServices.init
(NetworkModelsServices.java:215) 
hostname arc_builder[22051]: at com.q1labs.semsources.filters.arc.ArcBuilder.init(ArcBuilder.java:164)
hostname arc_builder[22051]: at com.q1labs.semsources.filters.arc.ArcBuilder.init(ArcBuilder.java:235)
hostname arc_builder[22051]:     ... 5 more
hostname arc_builder[22051]: 09/04/2017 22:06:18 22052 arc_builder error: Cannot load daemon


 

Resolving The Problem

Administrators who experience the issue reported in APAR IJ00838 can work with QRadar Support to increase their memory allocation for the arc_builder component.
  1. Log in to the QRadar Console.
  2. To verify for OutOfMemoryError messages in the deployment, type:
    /opt/qradar/support/all_servers.sh -C -k zless /var/log/qradar.log | grep OutOfMemoryError| tee IJ00838.txt

    This command will check/var/log/qradar.log on all managed hosts in the deployment and export a log file to IJ00838.txt.
  3. Review the output of IJ00838.txt to confirm the issue in your deployment.
  4. Open a case with QRadar Support to request assistance to optimize memory for arc_builder components that report OutOfMemoryError messages. 
  5. In your case, provide the following information:
    • Your QRadar version from Help > About.
    • A log export or attach the file IJ00838.txt to your case.
    • A contact number where you can be reached or an updated email so we can contact you in case our records are out of date.

      Results
      A QRadar Support representative will contact you to assist with a resolution for APAR IJ00838.


 

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"QRadar Risk Manager, arc_builder","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.2.8;7.3.0","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
20 February 2019

UID

ibm10872438