• No replies
1 Post

Pinned topic Max real bandwidth between one client and GPFS NSD servers

‏2013-04-29T15:44:59Z | balancing double fdr gpfs ib load port

I need to know how much GB/s I can read from a GPFS client.

I want to read at least 6 GB/s from one client on 2 NSD serveurs. We suppose that the 2 NSD servers are well configured, and each one can provide 4 GB/s (the both provide 8 GB/s in total)
We have Mellanox FDR infiniband network at 5700 MB/s (real) on each link.
Everybody is linked to the same Mellanox FDR 56Gb/s IB switch.
I would like to know "which is the biggest read bandwidth you reached in lab on one client"

My main problem is the FDR links which is limited to 5,7 Go/s
That's why I want to use 2 IB port on the GPFS client, with the special options in mmchconfig :

verbsPorts="mlx4_0/1 mlx4_0/2"
verbsPorts="mlx4_0/0 mlx4_1/0"
Are we allowed to use 2 different IB fabric ? (to have 2 differents switch to avoid a SPOF) Or the 2 ports have to be on the same Infiniband fabric ?

Do you think I can read 6 GB/s from a GPFS client using 2 IB ports ?
Does GPFS balance the load on the 2 Infiniband ports ?