IBM Support

QRadar Event Forwarding has sent events to storage

Troubleshooting


Problem

We are not receiving many events and are seeing notifications for Performance Degradation.

Symptom

You receive a notification with the following:
{Date} {Time} {loopback} [[type=com.eventgnosis.system.ThreadedEventProcessor][parent={HOSTNAME}:ecs-ec/EC/Forward/Forwarded_Event_Filter]] com.ibm.si.ec.filters.ForwardedFilter: [WARN] [NOT:0080004101][{Offending Host}/- -] [-/- -]Event Forwarded Filter has sent a total of {N} event(s) directly to storage. {N} event(s) have been sent in the last 60 seconds. Queue is at {N} percent capacity.

Cause

You are trying to forward more than 10K EPS.

Resolving The Problem

Before you begin
This solution is not persistent. The "number" parameter is overwritten during an upgrade per design. The number value needs to be reapplied after each upgrade.
Increase queue size for Forwarded_Event_Filter:
  1. Back up the existing files: 
    mkdir -p /store/ibm_support/6471605
cp -p /opt/qradar/conf/EC.xml /store/ibm_support/6471605/
cp -p /opt/qradar/conf/templates/configservices/EC.vm /store/ibm_support/6471605/
cp -p /store/configservices/deployed/LOCALSET/EC.xml /store/ibm_support/6471605/
  2. Edit /opt/qradar/conf/EC.xml, and change the "EVENT STACKS FOR THE COLLECTOR" for the "ForwardedEventFilter" "Number": 
    <!--    EVENT STACKS FOR THE COLLECTOR -->
<stack disabled="False" objectId="Forward" stdout="Processor2">
<filter objectId="Forwarded_Event_Filter" type="ForwardedEventFilter">
<parameter type="DestinationName">Processor2</parameter>
<parameter type="DestinationName">TrafficAnalysis1</parameter>
<parameter type="DestinationName">TCP_TO_VIS</parameter>
<parameter type="Name">7</parameter>
<!--  Queue Size --><parameter type="Number">10000</parameter>
    Change the "Number" parameter to use 40000:
    <!--  Queue Size --><parameter type="Number">40000</parameter>
  3. Type the command: 
    cp /opt/qradar/conf/EC.xml /store/configservices/deployed/LOCALSET/EC.xml
  4. Type the command: 
    chown root:root /opt/qradar/conf/EC.xml
  5. Type the command: 
    chmod u+rw,g+r,o+r /opt/qradar/conf/EC.xml
  6. Type the command 
    chown nobody:nobody /store/configservices/deployed/LOCALSET/EC.xml
  7. Type the command: 
    chmod u+rw,g+r,o+r /store/configservices/deployed/LOCALSET/EC.xml
     
    • Important:
      1. Restarting hostcontext results in services being restarted. While services are restarting, event processing stops until services restart. Scheduled reports that are in-progress need to be manually restarted by users. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
      2. Restarting hostcontext does not stop ecs-ec-ingress and incoming events are not interrupted.
  8. Restart hostcontext by using the command: 
    systemctl restart hostcontext
  9. The same value change needs to also occur in /opt/qradar/conf/templates/configservices/EC.vm or during a Deploy Full Configuration your changes are not persistent. This file is different so you are not able to copy it from the change you already made.

    Under: 
    <!--    EVENT STACKS FOR THE COLLECTOR -->
    Change: 
    <parameter type="Number">10000</parameter>
    to: 
    <parameter type="Number">40000</parameter>

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"TS006054604","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.2"}]

Document Information

Modified date:
20 July 2021

UID

ibm16471605

Page Feedback