Troubleshooting

Resolution or action

1. Check the output of mmvdisk rg list --rg <recovery_group_name> to see whether the recovery group is properly resized. If it is, continue with the next steps of the MES process.
2. If it was not properly resized, attempt to complete the log format by using the next command.

   mmvdisk rg create --complete-log-format

The following error message is displayed when you try to use SSH from one node to another, or when you run a command that requires SSH connections:

sshkey_read: Ed25519 keys are not allowed in FIPS mode

Run the following command in all affected nodes, where FIPS is enabled:

sed -i '/ed25519/d' /root/.ssh/known_hosts*

When you run the mmlsfirmware --type storage-enclosure command, the following error is displayed:

/usr/lpp/mmfs/bin/mmremote[8067]: lsFirmware[7565]: mmgetpdisktopologylite: line 2938: /usr/lpp/mmfs/bin/diskman/lsjbofnet: not found
mmlsfirmware: Command failed. Examine previous error messages to determine cause.

When you update a 5149-F48 server offline from both canisters, the secondary PSU might not be updated. During the service, LED illuminates becasue of this issue. The normal operation of the PSU has no impact. The following error message might be displayed:

[E] Error sg1 updating firmware for enclosure.

If the cluster is offline, then use ipmitool power cycle and run the update offline.

essrun update --offline

Resolution or action

1. Stop the existing bridge.

   virsh net-destroy ess_vm_host_only_network

2. Undefine the bridge.

   virsh net-undefine ess_vm_host_only_network

3. Create bridge with the latest DHCP reservation information.

   virsh net-define /opt/ibm/ess/tools/conf/network_host_only_kvm_dev_esstility.xml

4. Edit the XML.

   virsh net-edit ess_vm_host_only_network

5. Add the MAC address from the RAS interface to the EMSVM. An example is as follows:

   <ip address='10.23.16.1' netmask='255.255.255.248'>
    <dhcp>
    <range start='10.23.16.1' end='10.23.16.6'/>
    <host mac='52:54:00:b2:57:2e' name='EMSVM-23E' ip='10.23.16.2'/>
    <host mac='52:54:00:b2:57:2d' name='PROTOCOL_F48' ip='10.23.16.3'/>
    </dhcp>
    </ip>

6. Start the bridge.

   virsh net-start ess_vm_host_only_network

7. Set to autostart.

   virsh net-start ess_vm_host_virsh net-autostart ess_vm_host_only_networkonly_network

8. Verify whether the EMSVM RAS IP is 10.23.16.2 and start deployment from the GUI.

Run the following command to illuminate the associated enclosure LEDs. The blue enclosure LED illuminates and flashes for 60 seconds, and then turns off.

essjbofctl -i mgmt -j <JBOF S/N> -I -l Bottom  

[E] Can not determine slothandle to power drive /dev/nvmeXnY on/off

mmvdisk pdisk replace --rg <rg> --pdisk <pdisk name>

For example:

mmvdisk pdisk replace --rg rg1 --pdisk e1s5

1. From both I/O canisters check connectivity to the attached IBM Storage Scale System 5149-F26 storage enclosures.

   essjbofctl -i auto -t JX

   The JX is the JBOF config; for example, J1, J2.

2. Address errors if any. Reestablish connections to IBM Storage Scale System 5149-F26 storage enclosure that is attached to drives.

   essjbofctl -I auto -t JX -d -c

3. Verify the topology.

   mmvdisk server list --nc <building block node class> --disk-topology

4. Rerun the initial failing step after topology is verified.

Pdisk health check shows that the drive path is missing.

mmvdisk pdisk list --rg all --not-ok

1. Verify that corresponding DA is otherwise healthy and no rebuild operation is in progress.

   mmvdisk rg list --rg <RG name> --da --fault-tolerance

2. Power cycle the drive to reinitiate path discovery. Contact IBM Support for guidance on how to properly power cycle the drive.

An example of the error when you attempt to upgrade a single JBOD in a 5147-092:

mmchfirmware --type storage-enclosure --update-id PREVIOUS --serial-number 789A39W

The following error was displayed:

essio1-hs: [E] Storage enclosure 789A39W not found.
[17:20:07.829157999] /usr/lpp/mmfs/bin/mmchfirmware <- chEnclFirmwareVer2 rc:1
[17:20:07.829914727] /usr/lpp/mmfs/bin/mmchfirmware gpfs is up, putting pdisk paths back in service
mmchfirmware: Command failed. Examine previous error messages to determine cause.

1. To check whether the system has drives with degraded speed run the following:

   for i in $(nvme list -v | grep pcie | grep SAMS | awk '{print $NF}'); do  nvme list -v | grep pcie | grep $i | 
   awk '{printf $2" "}'; nvme list -v | grep pcie | grep $i | awk '{print $(NF-2)}' | xargs -i lspci -vvs {} 
   | grep LnkSta: ; done | grep -v 16GT

   A sample output is as follows:

   
   S7ZDNG0Y100024         LnkSta:    Speed 2.5GT/s (downgraded), Width x2 (ok)
   S7ZDNG0Y100038         LnkSta:    Speed 2.5GT/s (downgraded), Width x2 (ok)
   S7ZDNG0Y100023         LnkSta:    Speed 2.5GT/s (downgraded), Width x2 (ok)
   S7ZDNG0Y100051         LnkSta:    Speed 2.5GT/s (downgraded), Width x2 (ok)
   

2. Make sure that all drives are at 16 GT/s, otherwise, run the recovery steps:
   1. Power cycle the drive to reinitiate the link training.
   2. Reboot the node to reinitiate link training.
   3. Run recovery script to retrain the link. For example,

      /opt/ibm/ess/tools/samples/setlnkspd.sh 47

      A sample output is as follows:

      Slot: 47, target pcie address: d4:00.0, port pcie address: ca:09.0
             Link cap: 0a7b7904, Link Speed: 3, Link width 2
             Link cap 2: 021e0003
            Setting link cap 2 of slot 47 with 021E0004
            Trigger link training of slot 47? (y/n) y
            Triggering link control with 0060
      

You might see the following error when you enable SELinux on IBM Storage Scale System Utility Node VM:

[WARNING]: log file at /var/log/ess/6.2.3.0/essansible.json is not writeable and we cannot create it, aborting

Before a new protocol VM deployment from the GUI:

1. Verify whether any existing protocol VMs are present on the system.
2. If an existing protocol VM is found, remove it completely before you initiate the new deployment.
3. Clean the yum cache if needed by running yum clean packages.

1. Power off and on the drives.

   /opt/ibm/ess/tools/samples/drives_off_on

2. Enroll QLC drives (on one of the canisters).

If more issues or concerns, contact IBM Support.

1. Use underlying disk replacement commands.
2. Use the GUI to replace drives.

For more issues or concerns, contact IBM Support.

mmvdisk: (mkrginput) Only the format of 4K+0 or 512+0 is supported.

If two users load the deployment GUI portal for the same system and attempt to deploy without refreshing the page, the second deployment does not detect the ongoing progress of the first. As a result, the second deployment overwrites the first, and the first deployment stops. Any subsequent refresh of the GUI portal displays only the progress of the second deployment. This issue occurs only if both users reach the configuration step and click Submit to start deployment without refreshing the page beforehand.

To avoid this issue:

• Refresh the deployment GUI portal before deployment. A refresh checks for any ongoing deployment and prevents starting a new one if progress is detected.
• Make sure that only one user initiates the deployment at a time.

When a deployment fails, the GUI might display only a partial view of the essrun command logs.

To view the full essrun logs, manually access the deployment container and check the log file:

1. SSH into the deployment container host.

   ssh emsvm

2. Log in to the deployment container.

   podman exec -it IBMRCLCONNECT /bin/bash

3. Access the full log file.

   vim /var/log/ess/6.2.3.0/essansible.json 

TASK [/opt/ibm/ess/deploy/ansible/roles/checkenv : Check known hosts | Fail] ***
fatal: [utilityBareMetal]: FAILED! => changed=false
  msg:
  - Please check above output before proceeding to update.
  - Node has to be able to ssh to itself passwordless.
  - Manually fix by running 'ssh-keygen -R localhost ; ssh-copy-id localhost' in utilityBareMetal

To resolve the issue, run the following command on Utility Node:

ssh-keygen -R localhost ; ssh-copy-id localhost

# essstoragequickcheck

Invalid syntax error is as follows:

Traceback (most recent call last):
File "/opt/ibm/ess/tools/bin/essstoragequickcheck",
line 28, in <module>
from esstools import Listscsi_ess3k,
Listscsi_ess3200, Listscsi_ess3500, Listscsi_ess5k_ess5x,
Listscsi_s6k, Listadapter_ess3k, Listadapter_ess3200,
Listadapter_ess3500, Listadapter_ess5k,
Listadapter_ess5x, Listadapter_essutility,
Listadapter_s6k
File "/usr/lib/python3.6/sitepackages/
esstools/Listscsi_s6k.py", line 546
getSpeedCmd = "' %s /call /pall show | grep Expander
| awk \'\"\'\"\'{print $4}\'\"\'\"\'"%(storcli2) + " |
sort | uniq'" logger.info(getSpeedCmd)
^
SyntaxError: invalid syntax

1. Run the following command to remove the conflicting part of the script:

   sed -i 's/ *logger.info(getSpeedCmd)//' /usr/lib/python3.9/site-packages/esstools/Listscsi_s6k.py

2. Run the following command again:

   # essstoragequickcheck

   After the removal of the traceback, the command ran successfully.

# mmfsadm dump vdisk|grep -A 1 SpaceInfo

A sample output is as follows:

daSpaceInfoVersion 24:56376 daPhySpaceState Critical
used/total physical space 148086264/439482672 MiB utilization 0.3370

Where,

• 33.70% disk is used which means the space is in the Critical state. Run the following command to check how much space is available:

  grep utiliz /var/adm/ras/mmfs.log.latest

  A sample output is as follows:

  2024-12-09_20:50:05.911-0500: [C] Reset DA space utilization state threshold: Info:20
  Warning:22 Critical:30 OOS:35 OOSBGT:90

  OOS is at 35%, 1.3% less than OOS, and you can safely mount the file system. If the usage is at .349 or higher, delete more and rerun the emergency reclaim to free some more space.

  If the problem persists, call IBM Support.

In dmesg, you might see the following errors:

[Hardware Error]: Hardware error from APEI Generic 
 Hardware Error Source: 516
 [Hardware Error]: It has been corrected by h/w and 
 requires no further action
 [Hardware Error]: device_id: 0000:e1:00.0

After reconfiguration of the GUI, you might encounter the following error when you attempt to initialize GUI:

com.ibm.fscc.cli.CommandException: EFSSA0635A An error occurred while reading cluster information on host localhost

Resolution or action

• Run the following command from the management node:

  -rm /root/.pgpass
  -systemctl restart gpfsgui

• Initialize the GUI again.

Example:

# cat /proc/mdstat

A sample output is as follows:

 Personalities : [raid1]
 md2 : active raid1 nvmeXn1p3[2] nvmeYn1p3[0]
 133613440 blocks super 1.2 [2/2] [UU]
 bitmap: 0/1 pages [0KB], 65536KB chunk
 md3 : active raid1 nvmeXn1p5[2] nvmeYn1p5[0]
 768587904 blocks super 1.2 [2/2] [UU]
 bitmap: 6/6 pages [24KB], 65536KB chunk
 md1 : active raid1 nvmeXn1p2[2] nvmeYn1p2[0]
 8289280 blocks super 1.2 [2/2] [UU]
 bitmap: 0/1 pages [0KB], 65536KB chunk
 md0 : active raid1 nvmeXn1p1[2]
 2047936 blocks super 1.0 [2/1] [U_] << here is the missing mirrored partition.
 
 unused devices: <none>

In the example, you can see that nvmeYn1p1 is the missing element. To fix this manually, add the nvmeYn1p1 partition to /dev/md0 by running:

mdadm -a /dev/md0 /dev/nvmeYn1p1

# mmhealth node eventlog | grep temp_low_critical

A sample output is as follows:

2024-XX-XX 02:42:11.850411 EST temp_low_critical WARNING Temperature sensor drive_slot13_id94 has measured a temperature is less than the low critical value.

This error blocks a new key from being generated in the same NV slot.

• Option 1: The key is only mistakenly marked as in use; therefore, the same key can be reused. After the key is in use again, the rekey command can be used to change to the new key.
• Option 2: Contact IBM Support to reset the key.

# mmvdisk pdisk list --rg s6krw1 --not-ok

A sample output is as follows:

 recovery          declustered 
 group      pdisk  array     paths capacity free space FRU (type) state
 --------  ------  ------    ------ -----    -------- ----------   -----
 s6krw1    e1s24   DA1       1     52 TiB   51 TiB    03LG010      ok

This issue might resolve itself within an hour. If the issue is not resolved within an hour, complete the following steps:

1. Verify whether the system is in good condition and no arrays are rebuilding.
2. Reseat the affected drive.

If the issue is still not resolved, contact IBM Support.

• [ERROR] Unable to determine enclosure order. Check the HBA to enclosure cabling. [ERROR] Location 78F10GH-2 appears only on the sg369 path

• [ERROR] Enclosure 78F10G8 (IBM-ESS 5149-091, number undetermined):

• Adapter in PCI Slot: 2, index: c0 Enclosures connected: Enclosure e6 Serial 78F10GM Expected 91 Found 91 Connected to C3 at x4 22.5 Gb/s x12 Unknown

nvme list -v | grep NVMe | wc -l 40

esstoragequickcheck -N localhost 

A sample output is as follows:

[INFO] Enclosure 78L005D sees 40 disks (40 SSDs, 0 HDDs) [ERROR] GNR server disk topology: Scale 6000 F48 48 NVMe (match: 83/100)

Resolution or action

1. Reboot the canister.

   systemctl reboot

2. Complete AC power cycle the building block.
   1. Completely remove power from the IBM Storage Scale System 6000 chassis (unplug all the 4 PSUs).
   2. Wait for 5 minutes and plug power back to the PSUs.
   3. Wait for the canisters boot and check the drives again.
3. Scan the PCI bus again.

   echo 1 > /sys/bus/pci/rescan

You might need to power off or power on the affected slots. Contact IBM Support.

mmvdisk pdisk list --rg s6krg1 --not-ok

A sample output is as follows:

 recovery       declustered
 group    pdisk array paths capacity free space FRU (type) state
 ------  ------ ----- ----- -------- ---------- ---------  -----
 s6krg1  e1s22  DA1   3     52 TiB   12 TiB     03LG010    ok

The error might resolve itself within an hour. If the error is not resolved, use one of the following options:

• Verify that the system is in good condition and no arrays are rebuilding, then reseat the affected drive.
• Check your mounts, and verify whether you can reboot a canister without losing quorum. If you can, then reboot one canister. Wait until recovery is complete and check for the error. If the error is still unresolved, double check quorum and mounts again, and reboot that partner canister.
• Contact IBM Support, if the error is still not resolved.

1. Verify that you do not lose the quorum after you reboot the canister or a network connectivity does not change.
2. Change the link layer of the ports by using the essgennetworks command from the canister.

   essgennetworks -N localhost --change InfiniBand --devices /dev/mst/mt4129_pciconf1 --port P1

   essgennetworks -N localhost --change InfiniBand --devices /dev/mst/mt4129_pciconf1 --port P2

3. Reboot the canister.

   reboot

Example:

mmlsconfig autoload
autoload yes
autoload no

If the autoload setting is not as desired, use the mmchconfig command to change to the expected value.

mmchconfig autoload=yes

During an online update for IBM Storage Scale System 5000, you might encounter an error similar as follows:

TASK [/opt/ibm/ess/deploy/ansible/roles/updatenode : Set partner node as primary | gather backup] **************************
fatal: [essio2]: FAILED! => changed=true
  cmd: |-
    /usr/lpp/mmfs/bin/mmvdisk rg list --rg ess5k_essio1_essio2
    ess5k_essio2_essio1 --server -Y  | grep -i backup  | awk -F ':' '{print $8}'
  delta: '0:00:01.149195'
  end: '2024-XX-28 01:24:40.658240'
  failed_when_result: true
  msg: ''
  rc: 0
  start: '2024-XX-28 01:24:39.509045'
  stderr: '/bin/sh: line 1: ess5k_essio2_essio1: command not found'
  stderr_lines: <omitted>
  stdout: |2-

Run an offline update (as long as downtime can be scheduled).

1. Shut down GPFS on both I/O nodes.
2. Start the upgrade by using the --offline flag,

   essrun -N <IO nodeList> update --offline

If an offline upgrade is unacceptable, run an online upgrade.

1. Check the update.

   essrun update --precheck

2. Upgrade one node only.
3. From the RG group, move both recovery group names to the peer node.
   1. Select RG names that are hosted by the node and must be updated.

      mmvdisk rg list

   2. Check recovery groups status.

      mmvdisk rg change --rg <rg name> --active <peer node >

   3. Upgrade only one node (the one that is not hosting any RG).

      essrun -N <io node> update

   4. After one node online upgrade is successful, upgrade other nodes by completing all online upgrade steps.

Run the mmvdisk pdisk list --rg and use the output to determine whether an incorrect FRU type is seen.

• Correct FRU type

  mmvdisk pdisk list --rg rgs6k

  A sample output is as follows:

  
  recovery              declustered 
  group       pdisk     array        paths  capacity  free space   FRU (type)  state
  --------  ---------  -----------   -----  --------  -----------  ----------  ----
  rgs6k     e1s01         DA1        2      52 TiB    12 TiB       3LG010      ok 
  

• Incorrect FRU type

  
  recovery         declustered 
  group    pdisk   array         paths   capacity  free space   FRU (type)      state
  ------- ------  ----------     -----   --------  ----------  ---------------  -----
  rgs6k   e1s01   DA1              2      52 TiB   12 TiB      19.2 TBu FCM4 N  ok 
  

  The incorrect FRU does not impact on normal operation. However, a drive with incorrect FRU type needs to be replaced in the future, the normal repair operation might fail.

  If the repair fails, double check that you have the proper FRU type (03LG010). The FRU can be seen on the new drive label. If the FRU is incorrect, call IBM Support before continuing. If the FRU is correct, retry the repair by using the --force-fru option.

  mmvdisk pdisk replace –rg rgs6k --pdisk e1s01