IBM Support

QRadar: Server cannot restart correctly after upgrade due to modified fstab configuration

Troubleshooting


Problem

QRadar server does not restart correctly after an upgrade, this technical note covers one of the reasons this issue might occur, a customized fstab configuration.

Symptom

After the QRadar software is upgraded to a newer version, the server restarts but displays a "Login Error" message on the console window, administrators are unable to log in to the QRadar server.

Cause

The QRadar server is unable to restart correctly due to modifications saved to the fstab, as per the QRadar: Software update checklist for administrators page, all external storage, which is not /store/ariel or /store are required not to be mounted during the upgrade procedure.

Resolving The Problem

Administrators are advised to follow the QRadar: Software update checklist for administrators before starting the upgrade process.
Administrators should pay attention to the step "Unmount all external storage, which is not /store/ariel or /store".
To resolve this issue administrators are required to access the fstab on the Operating System, comment out custom added entries and restart the QRadar server to accomplish this the server must be restarted with the 'Factory re-install' option.
  1. Restart the QRadar server.
  2. At the first GRUB menu, press the down arrow key to stop the 5 second timer and highlight the option, Factory re-install. 
    Normal System
Factory re-install [QRadar 7.5.0 GA (Build 20211220195207)]
  3. Press the 'e' key to edit the entry.
  4. Scroll down to find the line that starts with "linux (loop)/isolinux/vmlinuz". 
    linux (loop)/isolinux/vmlinuz ks-hd:UUID
b-7fd28281c800:202160/ks.cfg action=reinstall root1vm=
    Note: Directories and UUIDs vary between systems.
  5. Edit the line and insert the string “rd.break” between “vmlinuz” and “ks=”.
    linux  (loop)/isolinux/vmlinuz  rd.break  ks=... 
    linux (loop)/isolinux/vmlinuz rd.break ks-hd:UUID
b-7fd28281c800:202160/ks.cfg action=reinstall root1vm=
  6. Press CTRL-x to reboot the server.
  7. When the server restarts, you are dropped at a command prompt. 
    Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


switch_root:/#
  8. From the switch_root prompt, run the following command to scan for physical volume groups. 
    lvm vgscan
    switch_root:/# lvm vgscan
   Reading all physical volumes. This may take a while... 
   Found volume group "rootrhel" using metadata type lvm2 
   Found volume group "storerhel" using metadata type lvm2 
switch_root:/#
  9. When the physical volumes are found, run the following command to activiate the volume groups. 
    lvm vgchange -ay
    switch_root:/# lvm vgchange -ay
   8 logical volume(s) in volume group "rootrhel" now active 
   2 logical volume(s) in volume group "storerhel" now active 
switch_root:/#
  10. Verify that the LVM volumes are visible in the /dev/mapper/ directory. 
    ls /dev/mapper/
    switch_root:/# ls /dev/mapper/
control      rootrhel-home    rootrhel-storetmp  rootrhel-varlog        storerhel-transient 
live-base    rootrhel-opt     rootrhel-tmp       rootrhel-varlogaudit
live-rw      rootrhel-root    rootrhel-var       storerhel-store
switch_root:/#
  11. Create a temporary directory. 
    mkdir -pv /tmp/root
    switch_root:/# mkdir -pv /tmp/root 
mkdir: created directory '/tmp/root' 
switch_root:/#
  12. Mount the Operating Systems root partition to the newly created temporary directory. 
    mount /dev/mapper/rootrhel-root /tmp/root
    switch_root:/# mount /dev/mapper/rootrhel-root /tmp/root/
[ 685.4192471 XFS (dm-2): Mounting V5 Filesystem
[ 685.562542] XFS (dm-2): Ending clean mount
switch_root:/#
  13. Change into the /tmp/root/etc directory. 
    cd /tmp/root/etc
    switch_root:/# cd /tmp/root/etc/ 
switch_root:/tmp/root/etc#
  14. Edit the fstab.
    vi fstab
  15. Change into Insert Mode by pressing 'i'. Place a '#' at the start of any customised or any customer added entries.
    For this example, I added a line.
    # Test Line 
    #
# /etc/fstab
# Created by anaconda on Tue Jan 4 17:38:47 2022
# Test Line
#
  16. To save your changes to the fstab, click Esc to enter command mode then type the following command to save and exit. 
    :wq
  17. Reboot the QRadar Server.
    reboot
    Let the server restart with the normal procedure.
Results
The server restarts correctly. You can log in with your root userid and the associated password.
Confirm that the changes were applied to the fstab file by viewing the file.
cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Jan 4 17:38:47 2022
# Test Line
#
Note: The changes that were made earlier adding the 'rd.break' entry to the line that starts with "linux (loop)/isolinux/vmlinuz" is automatically removed upon the restart of the server.
Note: When the QRadar server is restarting correctly, all custom entries in the /etc/fstab file should be mounted one at a time.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtdAAA","label":"Upgrade"}],"ARM Case Number":"TS010711561","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
08 June 2023

UID

ibm16959589

Page Feedback