IBM Support

QRadar: Deployments may experience an Event ingestion issue after the 4 October 2024 Auto Update is completed

News


Abstract

IBM has identified an issue where QRadar deployments may experience an outage of Event ingestion after the Auto Update released on 4 October 2024 is completed. Users who reporting this issue are not able to see events in Log Activity or new offenses being generated.


Content

Technical note updates

  • 4 October 2024 12:46 PM BST : A Flash notice was released to users to alert them of an Event ingestion issue . Users can apply the workaround documented in this flash notice when available.
  • 4 October 2024 03:55 PM BST: Flash notice updated with potential workaround
  • 4 October 2024 03:55 PM BST: A new Auto Update build has been posted to IBM Fix Central and auto update servers as QRADAR-QRAUTO-1728052152 on 04 October 2024 to resolve this issue.
  • 4 October 2024 10:33 PM BST: Additional steps added to address issues seen with syslog ingestion
  • 7 October 2024 04:30 PM BST: Updated instructions to re-add firewall rules to a database update.
  • 10 October 2024 05:15 PM BST: Updated text with known issues.
  • 29 October 2024 02:20 PM BST: Updated link to new TCP Syslog RPM
Update: The Known Issue DT398401 was resolved with the release of QRADAR-QRAUTO-1728052152 on 04 October 2024. Auto Update servers and Fix Central are updated to prevent this particular event ingestion issue from occurring in the deployment.
The Known issue DT398850 was resolved with the release of 7.5.0-QRADAR-PROTOCOL-TCPSyslog-7.5-20241009172754.noarch.rpm to fixcentral 28 October 2024, note, the workaround below is valid if the previously impacted protocol rpm was installed.

Urgency


Critical: QRadar deployments may experience an outage of Event ingestion after the Auto Update "QRADAR-QRAUTO-1727985725" released on 4 October 2024 is completed. The affected auto update bundle is removed from the auto update servers and IBM Fix Central.
The Auto Update caused the following two Known Issues:
1) Event collection disrupted as the geodata.conf file included in the Auto Update contained IPv6 addresses which could not be parsed. DT398401
2)The TCPSyslog protocol (PROTOCOL-TCPSyslog-7.5-20240917073309) included in the Auto Update in certain sceanrios prevented iptables opening port 514 causing event ingestion issues.  DT398850
 
NOTE: The new Auto Update is now available, you can install this AutoUpdate from the QRadar GUI directly or manually using the technote below.
The new Auto Update resolves the issue in DT398401 and has the impacted rpm in DT398850 removed.

https://www.ibm.com/support/pages/qradar-how-manually-install-qradar-weekly-auto-update-bundle

There is additional command line workarounds below to address both issues if you are unable to install the new Auto Update bundle at this time.

Affected products

QRadar SIEM Software Installations including QRadar on Cloud/QRadar SIEM SaaS

Note: QRadar on Cloud administrators should run Auto Updates to ensure they have the latest bundle. If you require any additional help please raise a case with QRadar Support.

Workaround

Procedure to resolve DT398401
  1. Use SSH to log in to the QRadar console as the root user.
     
  2. Type the following command the console to backup geodata.conf: 
    cp -p /opt/qradar/conf/geodata.conf /store/geodata.conf.bak
  3. Copy template to staging 
    cp -p /opt/qradar/conf/templates/geodata.conf /store/configservices/staging/globalconfig/geodata.conf
  4. Deploy will now be waiting in the GUI.
    Run a FULL Deploy, this will push the necessary files and restart ecs-ec and ecs-ep services as part of the deploy
     
  5. Wait for a few minutes and verify EPS is back to normal and offenses are being generated.

    Results
    After the deploy is completed and services restarted. QRadar should start to ingest events as normal.
NOTE: If after taking the above steps you are still not seeing syslog events on specific managed hosts, take the following steps.
Procedure to resolve   DT398850
  1. Use SSH to log in to the QRadar console as the root user.
    Then, for each host where syslog events are not being received:
  2. SSH to the host where events would be ingested
  3. Run this command to check if iptables has the appropriate rules for syslog:
    iptables -L -n | grep '\b514\b'

    If the rules are already present, the output should read: 
    # iptables -L -n | grep '\b514\b'
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:514
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:514
    If the rules are not already present, proceed with these steps:
  4. Run the following command to update the postgres database on the Console:
    psql -U qradar -c "update sensorprotocol set listenportkey='PORT' where id = 0;"
  5. The run a Full Deploy from the Console GUI
  6. When the deploy is running the following is displayed on the Console in /var/log/qradar.log  showing the firewall rule being re-added:
    Oct  7 10:21:08 ::ffff:127.0.0.1 [hostcontext.hostcontext]
[b6922aa7-88db-4c40-b946-ad33dc20c694/SequentialEventDispatcher]
com.q1labs.configservices.config.localset.iptables.EventCollectorIpRulesDelegate:
[INFO] [NOT:0000006000][x.x.x.x/- -] [-/- -]Adding firewall rule on host: x.x.x.x, protocol: Syslog, port: 514,
transProtocol: UDP, rule: -A QChain -m udp -p udp --dport 514 -j ACCEPT
Oct  7 10:21:08 ::ffff:127.0.0.1 [hostcontext.hostcontext] [b6922aa7-88db-4c40-b946-ad33dc20c694/SequentialEventDispatcher]
com.q1labs.configservices.config.localset.iptables.EventCollectorIpRulesDelegate:
[INFO] [NOT:0000006000][x.x.x.x/- -] [-/- -]Adding firewall rule on host: x.x.x.x, protocol: Syslog, port: 514,
transProtocol: TCP, rule: -A QChain -m state --state NEW -m tcp -p tcp --dport 514 -j ACCEPT
     
    Results
    After the deploy is completed and services restarted. QRadar should start to ingest syslog events as normal.
We apologize for any inconvenience due to this issue. If you have questions about the contents of this technical note, contact QRadar Support.

- QRadar Support

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtDAAQ","label":"Auto Update"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.0;7.5.0"}]

Document Information

Modified date:
29 October 2024

UID

ibm17172215

Page Feedback