Flashes (Alerts)
Abstract
Update IBM ESS 3000 boot drive firmware to FW1361 to prevent early failure of the boot drive.
Content
Systems Affected:
All release levels of ESS 3000 containing boot drive versions FW1236, FW1255 and FW1347. The boot drive firmware versions of each canister can be determined by using the following command. The command needs to be run from the canister you are working with to apply this firmware efix.
/opt/ibm/ess/tools/bin/ess3kfwLoader.sh -v | grep BOOTDRIVE
BOOTDRIVE A: FW1347
BOOTDRIVE B: FW1347
Boot drive firmware version FW1361 contains the necessary efix.
Cause and Symptoms:
Due to a defect in the boot drive firmware, it can cause the M.2 boot drive to reach the threshold of wear life warning faster than intended and trigger the “bootdrive smart failed” error event in ESS 3000.
The “bootdrive smart failed” error event serves as a warning to signify the need fora proactive replacement of a boot drive even though it is still fully functional. The recommendation is to schedule the proactive replacement within a month after the warning is reported at a customer’s earliest convenience.
Upgrade Procedure:
Follow the instructions to upgrade the boot drives to FW1361. This must be carried out for both canisters of each ESS 3000.
NOTE:
Do not copy or paste any command string as provided here, manually type the command to avoid any unintended modification by a document tool.
1.) Check system health and address pending issue first, especially any issue related to a boot drive in an ESS 3000 node.
mmhealth node show -N ess_x86_64-–unhealthy –-verbose
Example1:
[root@fab3a ~]# mmhealth node show -N ess_x86_64--unhealthy --verbose
Node name: fab3a-eth
Node status: HEALTHY
Status Change: 2021-05-12 18:05:36
Component Status Status Change Reasons
---------------------------------------------------------------------------------------------------------------
Node name: fab3b-eth
Node status: HEALTHY
Status Change: 2021-05-12 18:05:52
Component Status Status Change Reasons
----------------------------------------------------------------------------------------------------------------
Example2:
[root@ess3k3a ~]# mmhealth node show -N ess_x86_64--unhealthy –-verbose
Node name: ess3k3a-hs.gpfs.ess
Node status: DEGRADED
Status Change: 2021-04-18 11:10:45
Component Status Status Change Reasons
------------------------------------------------------------------------------------------------------------------------------------------------------------
NATIVE_RAID DEGRADED 2021-04-19 01:27:22 drive_firmware_wrong(78E016N, 78E016N,
78E016N)
ENCLOSURE DEGRADED 2021-04-18 11:10:45 drive_firmware_wrong(78E016N, 78E016N,
78E016N)
78E016N DEGRADED 2021-04-18 11:10:45 drive_firmware_wrong(78E016N, 78E016N,
78E016N)
Node name: ess3k3b-hs.gpfs.ess
Node status: DEGRADED
Status Change: 2021-04-18 11:05:11
Component Status Status Change Reasons
--------------------------------------------------------------------------------------------------------------------------------------------------------------
NATIVE_RAID FAILED 2021-04-18 11:10:26 bootdrive_smart_failed(78E016NB),
drive_firmware_wrong(78E016N, 78E016N, 78E016N)
CANISTER/SERVER FAILED 2021-04-18 11:05:10 bootdrive_smart_failed(78E016NB)
ENCLOSURE DEGRADED 2021-04-18 11:10:26 drive_firmware_wrong(78E016N, 78E016N,
78E016N)
78E016N DEGRADED 2021-04-18 11:10:26 drive_firmware_wrong(78E016N, 78E016N,
78E016N)
[root@ess3k3a ~]#
2.) Verify that the boot drives are in good health. Run the following command to obtain boot drive health from each canister of an ESS3000.
Contact IBM Support if summary states are not all optimal as seen in the example provided
tsplatformstat -m | mmyfields state status summary mountpoint
Example:
Example:
# tsplatformstat -m | mmyfields state status summary mountpoint
active [UU] optimal /serv
active [UU] optimal /var/log
active [UU] optimal /var
active [UU] optimal /home
active [UU] optimal /vpd
active [UU] optimal /tmp
active [UU] optimal /
active [UU] optimal /boot
tsplatformstat -b
Example:
# tsplatformstat -b
tsplatformstat:bootdrive:HEADER:model:serialNumber:firmwareVersion:capacity:selfAssessment:temperature:location: tsplatformstat:bootdrive:0:SRM2S86Q800GQT51IM01LL44701LL447IBM:S01LL447Y0DP98K00016:FW1347:800166076416:PASSED:055:100:ata-2: tsplatformstat:bootdrive:0:SRM2S86Q800GQT51IM01LL44701LL447IBM:S01LL447Y0DP9CP0000T:FW1347:800166076416:PASSED:055:100:ata-3:
3.) Download the firmware rpm from IBM Fix Central:
The following are the selection fields to use:
https://www.ibm.com/support/fixcentral/
> Find Product
> Product Selector: IBM Elastic Storage Server (ESS)
> Installed Version: All
> Installed Version: All
> Platform: Linux 64-bit,x86_64
Under ESS Firmware menu, download the ESS_FIRMWARE-6.0.0.11-x86_64-Linux.tgz to the "/home/deploy" directory in EMS node and extract the firmware rpm as follows:
cd /home/deploy
ls -l ESS_FIRMWARE-6.0.0.11-x86_64-Linux.tgz
date; tar zxvf ESS_FIRMWARE-6.0.0.11-x86_64-Linux.tgz
Example:
[root@ems1 deploy]# ls -l ESS_FIRMWARE-6.0.0.11-x86_64-Linux.tgz
-rw-r--r--1 root root 950084511 Aug 16 16:48 ESS_FIRMWARE-6.0.0.11-x86_64-Linux.tgz
[root@ems1 deploy]# date; tar zxvf ESS_FIRMWARE-6.0.0.11-x86_64-Linux.tgz
Mon Aug 16 16:51:04 EDT 2021
README
gpfs.ess.firmware-6.0.0-11.x86_64.rpm
4.) In each canister where the firmware rpm is to be applied, create the "/serv/efix" directory if not already created.
Copy the firmware rpm from the EMS node to "/serv/efix" of each canister.
5.) Shut down GPFS in the node where the FW upgrade is to be performed.
Ensure that quorum will be maintained with the remaining active nodes during shutdown to prevent any loss of access.
6.) ssh to the canister/node that is being updated and then update the firmware by removing the existing firmware rpm and installing the newly downloaded rpm.
rpm -e gpfs.ess.firmware
rpm -ivhgpfs.ess.firmware-6.0.0-11.x86_64.rpm
NOTE: gpfs.ess.firmware-6.0.0-11.x86_64.rpm is the expected FW rpm to apply for this FLASH.
7.) Verify again boot drive firmware version contained in the package and confirm an update is required:
/opt/ibm/ess/tools/bin/ess3kfwLoader.sh -c bootdrive -q /usr/lpp/mmfs/updates/latest/firmware/enclosure/ess3kplatfw.1111B.tar
NOTE: ess3kplatfw.1111B.tar is the expected drive FW package to apply for this FLASH
Example:
# /opt/ibm/ess/tools/bin/ess3kfwLoader.sh -c bootdrive -q /usr/lpp/mmfs/updates/latest/firmware/enclosure/ess3kplatfw.1111B.tar
BOOTDRIVEA: Current version (FW1347) does not match target version (FW1361)
BOOTDRIVEB: Current version (FW1347) does not match target version (FW1361)
Total mismatched level components: 2
8.) Verify current boot drive firmware level:
/opt/ibm/ess/tools/bin/ess3kfwLoader.sh -v | grep BOOTDRIVE
Example:
# /opt/ibm/ess/tools/bin/ess3kfwLoader.sh -v | grep BOOTDRIVE
BOOTDRIVEA: FW1347
BOOTDRIVEB: FW1347
9.) Perform the boot drive fw update:
/opt/ibm/ess/tools/bin/ess3kfwLoader.sh -c bootdrive -u /usr/lpp/mmfs/updates/latest/firmware/enclosure/ess3kplatfw.1111B.tar
Example:
# date;/opt/ibm/ess/tools/bin/ess3kfwLoader.sh -c bootdrive -u /usr/lpp/mmfs/updates/latest/firmware/enclosure/ess3kplatfw.1111B.tar
Mon Aug 16 17:48:13 EDT 2021
BOOTDRIVEA: Update -current FW1347to version FW1361
BOOTDRIVEA: Successful upgrade of boot drive BOOTDRIVEA!
BOOTDRIVEB: Update -current FW1347to version FW1361
BOOTDRIVEB: Successful upgrade of boot drive BOOTDRIVEB!
Upgrade of bootdrivea bootdriveb firmware(s) successful!
10.) Validate boot drive firmware levels after the upgrade:
/opt/ibm/ess/tools/bin/ess3kfwLoader.sh -v | grep BOOTDRIVE
Example:
# /opt/ibm/ess/tools/bin/ess3kfwLoader.sh -v | grep BOOTDRIVE
BOOTDRIVEA: FW1361
BOOTDRIVEB: FW1361
11.) Start GPFS and verify all nodes become 'Active', then repeat this firmware upgrade procedure for all other canisters that require the firmware efix.
Note:
- By design, this boot drive FW in this efix is expected to be preserved in a future upgrade to an ESS V6.1.1.2 or later version.
- Any future installation to an ESS 6.1.1.1, 6.0.2.x, or older release in an ESS 3000 will require a re-application of this efix as such code installation will remove the changes applied by this efix.
- The firmware RPM in the efix does include NVMe drive firmware that is newer than what is included in an ESS 6.0.1.2 or older installation. After the boot drive FW efix isapplied in such installation, “drive firmware wrong” error events would be reported because of the newer NVMe drive firmware level. The recommendation is to also apply the NVMe drive firmware included in the efix.
- In case updating NVMe FW is not desirable, please engage IBM support for assistance.
[{"Type":"SW","Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STHMCM","label":"IBM Elastic Storage Server"},"ARM Category":[{"code":"a8m50000000KzfKAAS","label":"Disk Errors"}],"Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]
Was this topic helpful?
Document Information
Modified date:
19 August 2021
UID
ibm16482437