developerWorks > IBM developer solutions > Forums > General Parallel File System (GPFS) >

GPFS Interconnects question concerning gpfsclustersfaq doc;


Search for within

My developerWorks

Welcome, Guest

	Sign in or register

Reply to this Thread

Post New Thread

Watch list

Forum settings

Help

	Replies: 3 - Pages: 1 - Last Post: Nov 10, 2009 2:35 PM Last Post By: dlmcnabb	Threads: [ Previous \| Next ]

cmack

Posts: 5
Registered: Jan 20, 2009 02:40:17 PM

GPFS Interconnects question concerning gpfsclustersfaq doc;
Posted: Nov 03, 2009 10:03:59 AM

Good morning!

This question is in concern of page 16 of the gpfsclustersfaq doc; question 3.5, what interconnects are support for GPFS daemon-to-daemon communication in a GPFS Cluster?

I see the answers given; however, I'd like a little expansion on the answers. Specifically, I am wanting to know;

1) Can I mix IB, Single GbE, and 10GbE interconnects in a single cluster with either Linux or AiX? Can an NSD serve out to clients via Eth and IB, or would I need to choose one or the other interconnect per NSD and have some NSD's server out eth, and others serve out IB?

2) Can I mix IB between AiX and Linux in a single GPFS Cluster( No Windows nodes). Same follow-up; Can an NSD serve out to clients via Eth and IB, or would I need to choose one or the other interconnect per NSD and have some NSD's server out eth, and others serve out IB?

3) Can I have some AiX and Linux nodes on Ethernet, and some Linux only Nodes on IB? Same follow-up; Can an NSD serve out to clients via Eth and IB, or would I need to choose one or the other interconnect per NSD and have some NSD's server out eth, and others serve out IB?

4) And like this thread here; https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14219110&#14219110 I too would be interested to see if there is a paper or other documentation concerning IB/RMDA as it concerns GPFS. As is, the documentation I have seen thus far is fairly light.

Thanks so much for your help. Have a great day!

dlmcnabb

Posts: 504
Registered: Mar 01, 2005 12:11:30 PM

Re: GPFS Interconnects question concerning gpfsclustersfaq doc;
Posted: Nov 05, 2009 02:27:49 PM

in response to: cmack's post

Use the most general external adapter when creating the cluster. Then if you specify the "subnets" configuration variable, you can list several different subnets any node may be attached to. Whenever two nodes make a connection, they will check if they have a common subnet (in order listed), and use that instead of the external adapter.

GPFS does not care what the machine types are at each end of a connection. It just uses whatever TCP socket is available to make connections.

cmack

Posts: 5
Registered: Jan 20, 2009 02:40:17 PM

Re: GPFS Interconnects question concerning gpfsclustersfaq doc;
Posted: Nov 10, 2009 02:22:20 PM

in response to: dlmcnabb's post

So I think you said that interconnect rules are less stringent when utilizing IPoIB...the follow-up question would be; does GPFS utilize SDP so that I may see the benefit of rdma, memory windows, etc..etc... for high performance zero copy data transfers between cluster nodes?

If not, when? Any additional info or public facing docs would be a blessing. Thanks much!

dlmcnabb

Posts: 504
Registered: Mar 01, 2005 12:11:30 PM

Re: GPFS Interconnects question concerning gpfsclustersfaq doc;
Posted: Nov 10, 2009 02:35:38 PM

in response to: cmack's post

SDP was found to be insufficient for GPFS usage.

However, GPFS can use RDMA for the block data transfers from the NSD servers. See the FAQ for the RDMA requirements, and the verbsRdma and verbsPorts configuration variables. If both the NSD server and a client node have the appropriate RDMA capabilities, the blocks will be trasferred using that method, otherwise it will fall back to sending the blocks over TCP.

All other GPFS communications will use TCP in IPoIB mode.

Pages: 1

Back to Thread List

Threads: [ Previous | Next ]

Update your My developerWorks profile and start connecting with others.