Troubleshooting
Problem
The docker service will fail if a bad line is added into the /opt/qradar/conf/iptables.pre file. If the apps are running on the console, the containers fail to start, and all apps become inaccessible in the UI. Even if there is an app host deployed, this can cause issues with the app framework and tomcat.
Symptom
This can occur whenever services restart, such as a system reboot, QRadar® version upgrade, or a full deploy. It can also cause severe performance issues in tomcat.
For example, during a patch, the upgrade_application section never completes.
You can identify the errors in the /var/log/setup<version>/patches.log:
Aug 31 12:56:22 2019: Aug 31 12:56:22 2019:[DEBUG](patchmode) 4 scripts ran for mode post_deploy.
Aug 31 12:56:22 2019: Aug 31 12:56:22 2019:[DEBUG](patchmode) returning 0; $VAR1 = {
'count' => 40,
'patchName' => '2019.14.0.20191006204340-2019140_patchupdate-2019.14.0.20191006204340'
};
Aug 31 12:56:22 2019: Aug 31 12:56:22 2019:[DEBUG](patchmode) called post-post deploy scripts.
Aug 31 12:56:22 2019: Aug 31 12:56:22 2019:[DEBUG](patchmode) Running: "/opt/qradar/bin/after_services_up.sh"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M100 266 100 266 0 0 1848
A dependency job for si-registry.service failed. See 'journalctl -xe' for details.
Aug 31 12:57:29 2019: Aug 31 12:57:29 2019:[ERROR](patchmode) Failed to run after_services_up.sh.
Cause
While the iptables_update.pl runs, it fails on the bad rule, and the iptables service consequently fails to start. This causes docker to fail to start, as the service depends on iptables.
Diagnosing The Problem
Check the docker and iptables status by running the following commands:
- systemctl status docker -l
docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/docker.service.d └─25_wants_containers.conf, 50-si-docker.conf, 75-wants_si_registry.conf Active: failed (Result: start-limit) since Wed 2020-08-31 13:54:04 ADT; 22min ago Docs: https://docs.docker.com Process: 15762 ExecStartPre=/opt/ibm/si/si-docker/bin/configure-docker-daemon.sh (code=exited, status=1/FAILURE) Process: 15218 ExecStartPre=/opt/ibm/si/si-docker/bin/configure-docker-network.sh pre (code=exited, status=0/SUCCESS) Oct 23 13:54:04 CONSOLE.FQDN systemd[1]: docker.service failed. Oct 23 13:54:04 CONSOLE.FQDN systemd[1]: start request repeated too quickly for docker.service Oct 23 13:54:04 CONSOLE.FQDN systemd[1]: Failed to start Docker Application Container Engine. Oct 23 13:54:04 CONSOLE.FQDN systemd[1]: docker.service failed.
- systemctl status iptables -l
Aug 31 11:37:21 CONSOLE.FQDN iptables.init[10675]: iptables: Applying firewall rules: iptables-restore: line 23 failed Aug 31 11:37:21 CONSOLE.FQDN iptables.init[10675]: [FAILED]
Resolving The Problem
You will need to edit the iptables rules in /opt/qradar/conf/iptables.pre
- Using SSH, log in to the QRadar® Console as the root user.
- Optional. Open an SSH session to the managed host that is receiving the data you want to block. (If not the Console appliance)
- Type the following command to edit the IPtables file:
Note: You can use VI, VIM, or any editor you choose. The IPtables configuration file is displayed.vi /opt/qradar/conf/iptables.pre - Enter your IPtables command:
-A INPUT -s <IP address> -p udp --dport 514 -j REJECT - Save your IPtables configuration, type the Esc key then :wq to save the changes and exit the editor.
- To update IPtables in QRadar, type the following command:
./opt/qradar/bin/iptables_update.pl - The change is applied to the appliance.
- To verify the new rules written in the /opt/qradar/conf/iptables.pre file are taken into account, you can perform the following command:
iptables -L -nNote: Verify that the host you want to block is listed in the iptables rules. - Verify the IPtables service is running by using the command: systemctl status iptables
Note: An incorrect rule could lock you out of the appliance you are adding the IPtables rule to.
Result: The applications upgrade completes with no failures.
Related Information
Document Location
Worldwide
[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.3.0;7.3.1;7.3.2;7.3.3;7.4.0"}]
Was this topic helpful?
Document Information
Modified date:
18 May 2021
UID
ibm16324709