IBM Support

QRadar: Replacement hard disk drive cannot rebuild and firmware state displays "JBOD"

Troubleshooting


Problem

When you replace a failed drive, the 930/530 RAID controller can set the drive into JBOD mode, which prevents a rebuild of the existing RAID virtual drive. This issue is due to a firmware problem. Administrators must set the status to unconfigured (good) state to ensure the drive can rebuild successfully.

Symptom

The replacement drive is not usable due to the current state of the drive in the enclosure. A review of the physical drive state can display System or JBOD, which prevents the RAID controller from successfully rebuilding the drive for the array.

Cause

QRadar M4, M5, and M6 hardware appliances with the latest Raid firmware might be affected by this JBOD state issue for replacement drives. Lenovo announced a RAID controller firmware fix is planned to prevent the issue later in 2022.

Environment

RAID adapters in QRadar M4, M5, and M6 hardware appliances are affected.

Diagnosing The Problem

To diagnose the replacement drive, confirm the current state of the physical and virtual drive with the ShowSummary command.
  1. Use a keyboard console or IMM to log in to the QRadar appliance with the replacement drive.
  2. To display the drive state, type:
    /opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary -a0
    The output displays the status. If the virtual drive displays 'Degraded' or 'Partially Degraded' and the state of the physical drive displays "System" or JBOD", you must correct the state of the drive.
    /opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary -a0
    ...
    
                    Connector          : Port 8 - 11<Internal>: Slot 9 
                    Vendor Id          : LENOVO  
                    Product Id         : ST8000NM001A   X
                    State              : Online
                    Disk Type          : SAS,Hard Disk Device
                    Capacity           : 7.276 TB
                    Power State        : Active
    
                    Connector          : Port 0 - 3<Internal>: Slot 2 
                    Vendor Id          : LENOVO  
                    Product Id         : HUS728T8TAL5200 
                    State              : System
                    Disk Type          : SAS,Hard Disk Device
                    Capacity           : 7.276 TB
                    Power State        : Active
    Storage
    
           Virtual Drives
                    Virtual drive      : Target Id 0 ,VD name 
                    Size               : 72.768 TB
                    State              : Partially Degraded
                    RAID Level         : 6
  3. To confirm the firmware state, type:
    ​/opt/MegaRAID/MegaCli/MegaCli64 -PDList -a0 | grep -B20 "Firmware" | grep "Enclosure Device\|Slot\|Firmware state\|--"
    ​
    The output displays the state and the enclosure and slot number.
  4. Identify any drives list the firmware state as JBOD.
    Enclosure Device ID: 134
    Slot Number: 0
    Firmware state: Online, Spun Up
    --
    Enclosure Device ID: 134
    Slot Number: 1
    Firmware state: Online, Spun Up
    --
    Enclosure Device ID: 134
    Slot Number: 2
    Firmware state: JBOD
    --
    
    
    Results
    If the firmware state displays JBOD, you are affected by the issue and must set the replacement drive to an unconfigured state.  It is important to record the enclosure ID and the slot number as this information is required to resolve the issue. In the example, the controller displays the disk in Slot 2, on Enclosure Device ID 134 in a JBOD state.

Resolving The Problem

Before you begin
Ensure you use the correct Enclosure Device and Slot number for the drive in a JBOD state. If you force a drive that is already online to the unconfigured good state, it might further degrade the virtual drive or take the virtual drive offline.

Procedure
To resolve the problem, change the drive state to unconfigured good. As the drive is configured as JBOD, it is likely assigned as a device in the OS and you might need to use the -Force option to set the state.
  1. Type the following command to change the state for the drive to 'unconfigured good':
    Important: The examples in this section use a drive in Enclosure 134, Slot 2 for adapter 0. The drive enclosure and slot values are likely different on your appliance. It is important you use the correct values in this procedure for the replacement drive in your QRadar appliance.
    /opt/MegaRAID/MegaCli/MegaCli64 -PDMakeGood -PhysDrv [134:2] -Force -a0
  2. Configure the replacement drive as a hot spare with the following command:
    /opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Set -PhysDrv [134:2] -a0

    Results
    The replacement drive is recognized as a hot spare and the RAID controller can rebuild the disk for the array. It might take a few minutes for the rebuild process to begin. The view the rebuild status type the following command:
    /opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ShowProg -PhysDrv [134:2] -a0

Preventing JBOD state issues for replacement drives

After you resolve the problem to allow replacement drives to be rebuilt, administrators can use the StorCLI utility to turn off JBOD mode. Disabling JBOD ensures the firmware state is not incorrectly set by the RAID controller on future drive replacements.
 
The StorCLI utility can likely be found in 1 or 2 locations in QRadar depending on Software version:
  • /opt/lenovo/lnvgy-utl-lxce-onecli/utilities/lsi/storcli
  • /opt/MegaRAID/storcli
You can use either command to set the JBOD state to off. Use the following command to show the JBOD state on controller 0. If the controller properties shows JBOD is on, we can then use the set command to turn it off.
  1. To confirm the controller properties, type:
    /opt/lenovo/lnvgy-utl-lxce-onecli/utilities/lsi/storcli /c0 show jbod
    The output confirms the JBOD orpoerty status.
    Controller Properties:
    =====================
    ----------------
    Ctrl_Prop Value
    ----------------
    JBOD      ON
    ----------------
  2. Configure the controller properties to disable JBOD on the appliance:
    /opt/lenovo/lnvgy-utl-lxce-onecli/utilities/lsi/storcli /c0 set jbod=off
    /opt/lenovo/lnvgy-utl-lxce-onecli/utilities/lsi/storcli /c0 set jbod=off force
  3. Verify the status of JBOD is set to off:
    /opt/lenovo/lnvgy-utl-lxce-onecli/utilities/lsi/storcli /c0 show jbod 
    The output displays the current properties.
    Controller Properties:
    =====================
    ----------------
    Ctrl_Prop Value
    ----------------
    JBOD      OFF
    ----------------
    
    Results
    After you confirmed JBOD is disabled, the procedure is complete. If you continue to experience issues or problems with this technical note, contact QRadar Support.

     

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSQKYT","label":"IBM QRadar xx48 Appliance"},"ARM Category":[{"code":"a8m0z000000cwtcAAA","label":"Hardware"}],"ARM Case Number":"TS008497375","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtcAAA","label":"Hardware"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
07 June 2022

UID

ibm16573915