Troubleshooting
Problem
ESS 5.3.4 now includes MLNX_OFED 4.6. A bug was found in the Mellanox driver on MLNX_OFED 4.6 that impacts only the Connect-IB adapter code path within the driver. This impacts both BE and LE ESS systems.The adapter will show up and the MLNX_OFED driver will configure the IB interfaces. However, when VerbsRDMA option is enabled and GPFS is started, it will display RDMA connection errors in the GPFS log (/var/adm/ras/mmfs.log.latest). GPFS then proceeds to shut down soon after.
The errors in the mmfs.log.latest will show that the card has failed to initialize. The Connect-IB adapter is unsupported for this ESS 5.3.4 release ONLY.
Symptom
ESS customers with VerbsRDMA enabled, during upgrade to ESS 5.3.4 will see one or more of the following messages displayed:
1) When running "gssinstall_<arch>"
***********************
This command is run to create/update the local repositories on the EMS. We placed a check inside that looks for the Connect-IB adapter. The script will check for a Connect-IB adapter and flag it and abort the install.
If Connect-IB detected:
[ERROR]: Unsupported Mellanox cards found in system.
[HINT] Check the installed card is supported or not.
[HINT] Mellanox Connect-IB cards detected on system which is not supported.
The script will fail to proceed with yum repository creation.
This command is run to create/update the local repositories on the EMS. We placed a check inside that looks for the Connect-IB adapter. The script will check for a Connect-IB adapter and flag it and abort the install.
If Connect-IB detected:
[ERROR]: Unsupported Mellanox cards found in system.
[HINT] Check the installed card is supported or not.
[HINT] Mellanox Connect-IB cards detected on system which is not supported.
The script will fail to proceed with yum repository creation.
***********************
2) When running "gssprecheck"
2) When running "gssprecheck"
******************************
This command is run to detect and advise against common install and upgrade issues.
[ERROR] Checking for unsupported Mellanox cards
[HINT] Check the installed card is supported or not.
[HINT] Mellanox Connect-IB cards detected on system which is not supported.
*****************************
This command is run to detect and advise against common install and upgrade issues.
[ERROR] Checking for unsupported Mellanox cards
[HINT] Check the installed card is supported or not.
[HINT] Mellanox Connect-IB cards detected on system which is not supported.
*****************************
3) When running "gss_updatenode"
****************************
This script updates the O/S (if applicable) and performs other tasks like installing Spectrum Scale. If Connect-IB is detected you will see the following message and the script will exit preventing the user from proceeding.
Detected Connect-IB cards on system which is not supported.
Upgrade aborted. Exiting...
This script updates the O/S (if applicable) and performs other tasks like installing Spectrum Scale. If Connect-IB is detected you will see the following message and the script will exit preventing the user from proceeding.
Detected Connect-IB cards on system which is not supported.
Upgrade aborted. Exiting...
*****************************
Resolving The Problem
Fix is available.
Customers that are affected may apply ESS V5.3.4.1 or later, available from Fix Central at :
Please contact IBM Service for support at http://www.ibm.com/planetwide/
Document Location
Worldwide
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STHMCM","label":"IBM Elastic Storage Server"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"5.3.4","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]
Was this topic helpful?
Document Information
Modified date:
06 August 2019
UID
ibm10887897