IJ21097: "IBV_MODIFY_QP INIT ERR 22" FOR 2ND PORT IF VERBSRDMACM ENABLED

APAR status

• Closed as program error.

Error description

• When verbsRdmaCm is enabled it will give the following
  error for the second port. mmfsadm dump verbs only shows
  1 port being used for each nsd server. One cannot utilize
  the second port.
  
  
  [E] VERBS RDMA error connecting to 192.168.1.1 (xxxxx...)
  on mlx5_2 port 1 ibv_modify_qp init err 22
  
  Reported In:
  
  Spectrum Scale 5.0.4.0
  

Local fix

• temporarily disable verbsRdmaCm
  

Problem summary

• The ports of 2nd (and later) IB adapter on the node which starts
   the verbs connection  might be mis-recognized
   as RDMA CM disabled ports, and fails to
  be connected. The  nodes that start the verbs
  connection are nsd clients if
  verbsRdmaSend=no, but they
  also could be other nodes if verbsRdmaSend=yes.
  You will see "ibv_modify_qp init err 22" error message in the
  mmfs.log file if it happens.
  

Problem conclusion

• Benefits of the solution:
  
  
  Fix the problem
  
  Work Around:
  
  None. But if RDMA-CM is not really needed in your environment,
  you can just disable it.
  
  Problem trigger:
  
  Users having
  
  Symptom:
  
  Performance Impact/Degradation
  
  Platforms affected:
  
  ALL Linux OS environments
  
  Functional Area affected:
  
  RDMA
  
  Customer Impact:
  
  High Importance
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SPEC SCALE STD

• Fixed component ID

  5737F33AP

Applicable component levels