IBM Support

QRadar: Services don't start after an upgrade due to QRadar booting to a previous kernel

Troubleshooting


Problem

QRadar patches install a new kernel version on the system. After the patch reboots the appliance, it boots to a previous kernel instead of the new one recently installed by the patch causing some of the services not to start.

Symptom

The journalctl of the IP tables service reports the following problem:
journalctl -u iptables | grep "iptables.init" -B 1 | tail -n 5

systemd[1]: Starting IPv4 firewall with iptables...
iptables.init[31958]: iptables: Applying firewall rules: iptables-restore: unable to initialize table 'filter'
iptables.init[31958]: Error occurred at line: 17
iptables.init[31958]: Try `iptables-restore -h' or 'iptables-restore --help' for more information.
iptables.init[31958]: [FAILED]
The error "unable to initialize table 'filter'" indicates that the right kernel is not used.

Environment

QRadar appliances after an upgrade or patch.

Diagnosing The Problem

In this technote, QRadar 7.4.3 FixPack4 InterimFix02 is used as an example. The administrators are advised to run the steps on their required version as the output values for the kernel might differ.
  1. Login as the root user to an unaffected appliance that runs the same version as the affected host.
    Note:     The unaffected appliance can be any managed host.
    1. Verify the current QRadar version by using the myver script.
        
      /opt/qradar/bin/myver -v | head -9
Product is 'QRadar'
Appliance is '3105'
Core version is '2020.11.4.20211113154131'
Interim fix number is '02'
Latest version is '2020.11.4.20211113154131'
Branded version is ''
External version is '7.4.3'
Branded latest version is ''
Release name is '7.4.3 FixPack 4'
    2. Verify the running kernel information with the uname command.
        
      uname -r
3.10.0-1160.49.1.el7.x86_64
  2. Login as the root user to the affected appliance.
    1. Verify the current QRadar version by using the myver script. The patch version (including interim fix) must match the unaffected appliance.
        
      /opt/qradar/bin/myver -v | head -9
Product is 'QRadar'
Appliance is '3105'
Core version is '2020.11.4.20211113154131'
Interim fix number is '02'
Latest version is '2020.11.4.20211113154131'
Branded version is ''
External version is '7.4.3'
Branded latest version is ''
Release name is '7.4.3 FixPack 4'
    2. Verify the running kernel information is older than the unaffected appliance with the uname command. 
        
      uname -r
3.10.0-1127.10.1.el7.x86_64
    3. Verify the kernel that is taking precedence at boot in the /boot/grub2/grub.cfg. The top entry is the one that takes precedence.
        
      grep "menuentry 'Red Hat Enterprise Linux Server" /boot/grub2/grub.cfg | head -n 2 | cut -d " " -f 1-9

Output Example:
menuentry 'Red Hat Enterprise Linux Server (3.10.0-1127.10.1.el7.x86_64) 7.6 (Maipo)' 
menuentry 'Red Hat Enterprise Linux Server (3.10.0-1062.7.1.el7.x86_64) 7.6 (Maipo)'
    4. Verify the kernel use at boot by default in the /boot/grub2/grubenv. The line with "saved_entry" is the default kernel.
        
      grep saved /boot/grub2/grubenv

Output Example:
saved_entry=Red Hat Enterprise Linux Server (3.10.0-1127.10.1.el7.x86_64) 7.6 (Maipo)
The previous example shows that the appliance that runs QRadar 7.4.3 FixPack4 InterimFix02 must run the kernel 3.10.0-1160.49.1.el7.x86_64, however, the affected appliance runs 3.10.0-1127.10.1.el7.x86_64 despite running the same QRadar patch version.

Resolving The Problem

In this technote, QRadar 7.4.3 FixPack4 InterimFix02 is used as an example. The administrators are advised to run the steps on their required version as the output values for the kernel might differ.
To resolve this problem, the administrator must reinstall the kernel rpm package provided by the patch.
 
  1. Login as the root user to the affected appliance.
    1. Verify the QRadar version.
        
      /opt/qradar/bin/myver -v | head -9
    2. Create a working directory.
        
      mkdir -p /store/IBM_Support/kernel_rpms/
  2. Login as the root user to an unaffected appliance that runs the same version as the affected host.
    Note: The unaffected appliance can be any managed host.
  3. Obtain the QRadar SFS file from the QRadar 101 page and copy it to the /storetmp partition.
  4. Mount the SFS file in the /storetmp partition by following the steps in the release notes.
    1. Create the /media/updates directory.
        
      mkdir -p /media/updates/
    2. Mount the patch file to the /media/updates directory.
        
      mount -o loop -t squashfs /storetmp/<QRadar patch file>.sfs /media/updates
  5. Use the scp command to copy the required files to the affected appliance.
    Note: This step requires SSH connectivity between the unaffected appliance and the affected appliance.
      
    scp /media/installer/repo/kernel* <Affected appliance IP>:/store/IBM_Support/kernel_rpms/
  6. Unmount the patch file in the unaffected appliance.
      
    umount /media/updates
  7. Login as the root user to the affected appliance.
  8. Reinstall the kernel rpm files recently copied from the unaffected appliance.
      
    cd /store/IBM_Support/kernel_rpms/
yum -y reinstall kernel*
  9. Update the grub entries.
      
    grub2-mkconfig -o /boot/grub2/grub.cfg
  10. Verify the kernel that takes precedence and the default kernel is updated with the recently installed kernel.
      
    grep "menuentry 'Red Hat Enterprise Linux Server" /boot/grub2/grub.cfg | head -n 2 | cut -d " " -f 1-9

grep saved /boot/grub2/grubenv
  11. Compare the previous output with the output in steps 2c and 2d in the "Diagnosing the Problem". The new kernel version must be at the top and listed as the default kernel.
  12. Reboot the affected appliance.
Results
After the affected appliance boots, the services now start without problems. If the services still fail to start, open a case with QRadar Support.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtdAAA","label":"Upgrade"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
21 April 2022

UID

ibm16572877

Page Feedback