Topic
  • 2 replies
  • Latest Post - ‏2013-06-26T11:03:10Z by RecercaUPF
RecercaUPF
RecercaUPF
7 Posts

Pinned topic useNSDserver=always not working in NSD servers on 3.5

‏2013-06-21T09:58:35Z |

Hi all,

We have 4 NSD servers connected each one using one SAS wire to 4 disk cabinets. Each cabinet has 4 LUNs/NSDs and each server is the main responsible of one NSD per cabinet.

 

File system   Disk name    NSD servers                                    
---------------------------------------------------------------------------
 scratch       scratch11 server1,server2,server3,server4
 scratch       scratch21 server1,server3,server4,server2
 scratch       scratch31 server1,server4,server3,server2
 scratch       scratch41 server1,server3,server2,server4
 scratch       scratch12 server2,server1,server4,server3
 scratch       scratch22 server2,server4,server1,server3
 scratch       scratch32 server2,server3,server1,server4
 scratch       scratch42 server2,server4,server1,server3
 scratch       scratch13 server3,server4,server1,server2
 scratch       scratch23 server3,server1,server4,server2
 scratch       scratch33 server3,server2,server4,server1
 scratch       scratch43 server3,server1,server4,server2
 scratch       scratch14 server4,server3,server1,server2
 scratch       scratch24 server4,server2,server3,server1
 scratch       scratch34 server4,server1,server2,server3
 scratch       scratch44 server4,server2,server1,server3

scratchXY means cabinet X, LUN/NSD/server Y

 

We are mounting the filesystem with -o useNSDserver=always on all nodes, both NSD servers and clients. And it works for the clients, but not for the servers.

Whenever a server writes a big file to the fs, it ignores the useNSDserver=always option and connects to the disks LOCALLY. We have only one SAS wire from each server to disk cabinet, meaning that each server connects only to ONE of the two controllers of each cabinet. These two things combined leads to constant changes of the preferred paths on the disk controllers, and the subsequent mail alert spamming.

 

Is there a way to fix this? We have defined those disks as devtype=dmm. Should "generic" do not cause this?

 

 

  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: useNSDserver=always not working in NSD servers on 3.5

    ‏2013-06-21T14:04:30Z  

    If a node is listed as one of the NSD servers (even though not the primary server), then it will do the IO locally.

  • RecercaUPF
    RecercaUPF
    7 Posts

    Re: useNSDserver=always not working in NSD servers on 3.5

    ‏2013-06-26T11:03:10Z  
    • dlmcnabb
    • ‏2013-06-21T14:04:30Z

    If a node is listed as one of the NSD servers (even though not the primary server), then it will do the IO locally.

    Thanks for your answer

    So this it is working as intended? That makes me sad.

    So, if the number of "server-to-disk" connections is limited (direct SAS wires, no switch involved), I have to choose between having a single-wire all-to-all network (and having those issues with the "jumping" controllers) or creating Building Blocks (and decrease the systems resilence to server failures)??

    Is it a choice of reliability vs work load in the servers?

     

    Updated on 2013-06-26T11:03:23Z at 2013-06-26T11:03:23Z by RecercaUPF