IBM Support

QRadar: Kernel 3.10.0-1127.EL7.X86_64 can cause XFS filesystem mount failures in QRadar 7.4.0 Fix Pack 3 (APAR IJ25612)

Troubleshooting


Problem

Administrators who upgrade to QRadar® 7.4.0 Patch 3 can experience a Red Hat kernel issue where appliances are unable to mount the filesystem or properly boot as documented in APAR IJ25612. Administrators can experience this issue on a per appliance basis. To assist users in identifying this issue, QRadar development has created an identification utility that can be run on appliances to identify potential issues.

Symptom

Administrators who install QRadar® 7.4.0 Fix Pack 3 can install the fix pack, but the appliance reboots fails with a number of XFS errors where the filesystem cannot be mounted. This issue is identified as APAR IJ25612 by QRadar Support.

Error message:
XFS (dm-0): SB strip unit sanity check failed
XFS (dm-0): SB validate failed with error -117. 
Screen capture:
image 4420
Figure 1: After the fix pack installation, an appliance reboot can fail the XFS filesystem check due to Red Hat kernel issues.

Cause

This issue is identified as a problem in Red Hat kernel 3.10.0-1127.el7.x84_64 as identified in Red Hat Bug Advisory: RHBA-2020:2355 - xfs: catch bad stripe alignment configurations.

Environment

QRadar® versions that contain kernel 3.10.0-1127.el7.x86_64:
  • QRadar 7.4.0 Fix Pack 3
  • QRadar 7.4.0 Fix Pack 2
  • QRadar 7.3.3 Fix Pack 3

Diagnosing The Problem

Before administrators upgrade to QRadar® 7.4.0 Fix Pack 3, they can run the IJ25612.tgz utility to identify appliances that can experience this kernel issue. The utility must be run on each QRadar appliance to determine if the host passes or fails the test. The utility identifies whether the minimum IO size exceeds the optimal IO size for devices with an optimal IO size greater than zero.

Procedure
  1. Download the support utility: IJ25612.tgz
  2. Copy the file to the QRadar Console.
  3. Confirm the MD5 / 256 sum for the file:
    MD5 - 248ed6814e9675947e1d5843d25607da  IJ25612.tgz
    SHA256 - 825fee9b8f871574bf73e809255ca44d55436678f3616a45498e9c8ec518b024  IJ25612.tgz
  4. To extract the file, type:
    tar zxvf IJ25612.tgz
  5. Set permissions on the file with the following command:
    chmod +x IJ25612.sh
  6. To copy this file to all QRadar appliances, type:
    /opt/qradar/support/all_servers.sh -kp /root/IJ25612.sh
  7. Use SSH to open a connection to each host in the deployment.
  8. Navigate to the /storetmp directory.
  9. Run IJ25612.sh.
    Note: You must run the utility on each QRadar appliance to diagnose any impacted hosts.
  10. Review the output to determine whether your appliance is impacted by IJ25612.

    Affected appliance output
    Failed checks indicate the appliance is impacted by the Red Hat kernel issue identified in IJ25612. If you upgraded to QRadar 7.4.0 Fix Pack 3, you can reboot the appliance and select the previous kernel version in the grub menu. If an appliance reports as affected, administrators can wait for a software resolution in the next fix pack to ensure you do not experience this issue.
    [root@qradar_examplehost tmp]# ./IJ25612.sh
            /                                           Failed check
            /var/log                                    Failed check
            /opt/qradar/bin/ca_jail/lib64               Failed check
            /opt/qradar/bin/ca_jail/lib                 Failed check
            /opt/qradar/bin/ca_jail/usr/share           Failed check
            /opt/qradar/bin/ca_jail/usr/lib64           Failed check
            /opt/qradar/bin/ca_jail/usr/bin             Failed check
            /opt/qradar/bin/ca_jail/bin                 Failed check
            /opt/qradar/bin/ca_jail/usr/lib             Failed check
            *****************************************************************
            **                         IJ25612                              **
            **  xfs filesystems found that will result in failure to mount  **
            **  and cause the QRadar appliance to fail to boot              **
            **  This is due to the use of kernel 3.10.0-1127.el7.x86_64     **
            **  as identified in the following note:                        **
            **  https://access.redhat.com/solutions/5075561                 **
            **                                                              **
            **          Don't apply QRadar 7.4.0 FixPack 3                  **
            **                                                              **
            *****************************************************************

    Not affected by IJ25612 output
    Administrators who see the following output message can proceed with the installation of QRadar 7.4.0 Fix Pack 3. Administrators must confirm that each appliance reports this error message before the upgrade to QRadar 7.4.0 Fix Pack 3.
    No xfs filesystems found that will hit IJ25612 a bug in kernel 3.10.0-1127.el7.x86_64
    You can proceed with the patch to QRadar 7.4.0 Fix Pack 3

Resolving The Problem

If the QRadar® appliance is booted into a previous version of the Red Hat kernel, the XFS file system can be mounted. If you upgraded your appliance and the XFS filesystem fails to mount after an upgrade to QRadar 7.4.0 Fix Pack 3, complete the following steps.
Procedure
Administrators must confirm that all appliances in the deployment mount the XFS filesystem after you upgrade and select the previous kernel version from the grub select menu.
  1. Reboot the QRadar® appliance.
  2. On the GRUB 2 boot screen, use the cursor and select Red Hat Enterprise Linux 3.10.0-1062.1.1.
  3. Press Enter to boot in to the previous kernel.
  4. Verify the appliance reboots and successfully mounts the XFS filesystem.
  5. Repeat this procedure for each appliance that experiences issue IJ25612.

    Results
    If you experience any issues, administrators can contact IBM QRadar Support for assistance.

Document Location

Worldwide

[{"Business Unit":{"code":"BU008","label":"Security"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000Gnc6AAC","label":"QRadar->Upgrade"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3.3;7.4.0"}]

Document Information

Modified date:
22 June 2020

UID

ibm16235774