Topic
  • 4 replies
  • Latest Post - ‏2013-12-06T10:07:20Z by perttu
perttu
perttu
3 Posts

Pinned topic How to direct IO from ethernet to FC?

‏2013-12-03T15:15:49Z |

Hi all,

I have configured both ethernet access (throw nsd-servers) and fibre channel access (npiv) to my GPFS -disks. When I mount the filesystems IO-traffic goes through  disks and HBA:s if the disks are available.  If the disks are not available the filesystems get mounted through nds-servers and IO-traffic goes through en-devices. Unfortunately I haven't find a non-disruptive way to redirect IO-traffic to go through disks instead of ethernet.

How do I change the traffic routing from ethernet to disk without unmounting and mounting?

  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: How to direct IO from ethernet to FC?

    ‏2013-12-03T16:19:31Z  

    If the disks become available after being missing, try running

    mmnsddiscover -a -N all

    This will first have all NSD servers check for devices coming back, and then all the client nodes will reevaluate whether they see disks locally or served by NSD servers.

     

    If devices are not visible on nodes fast enough when GPFS first comes up, then you may have to put some code in /var/mmfs/etc/gpfsready script to check that disks are available before letting GPFS start any mounts. This script blocks GPFS starting until it returns.

  • perttu
    perttu
    3 Posts

    Re: How to direct IO from ethernet to FC?

    ‏2013-12-05T14:11:07Z  
    • dlmcnabb
    • ‏2013-12-03T16:19:31Z

    If the disks become available after being missing, try running

    mmnsddiscover -a -N all

    This will first have all NSD servers check for devices coming back, and then all the client nodes will reevaluate whether they see disks locally or served by NSD servers.

     

    If devices are not visible on nodes fast enough when GPFS first comes up, then you may have to put some code in /var/mmfs/etc/gpfsready script to check that disks are available before letting GPFS start any mounts. This script blocks GPFS starting until it returns.

    Thank You for Your help.

    Unfortunately mmnsddiscover didn't do it :( mmlsnsd -m shows that disk
    is available locally but when I start writing to disk network traffic
    peeks and disk io stays flat.

    However, from what I've read change should be automatic:

    - mount -option useNSDserver=asneeded is default

    - failover shoud happen from local disk to nsd and vice verca

    Maybe I should try to throttle network and force failover that way?

    Updated on 2013-12-05T14:55:32Z at 2013-12-05T14:55:32Z by perttu
  • yuri
    yuri
    261 Posts

    Re: How to direct IO from ethernet to FC?

    ‏2013-12-06T02:59:58Z  
    • perttu
    • ‏2013-12-05T14:11:07Z

    Thank You for Your help.

    Unfortunately mmnsddiscover didn't do it :( mmlsnsd -m shows that disk
    is available locally but when I start writing to disk network traffic
    peeks and disk io stays flat.

    However, from what I've read change should be automatic:

    - mount -option useNSDserver=asneeded is default

    - failover shoud happen from local disk to nsd and vice verca

    Maybe I should try to throttle network and force failover that way?

    That's not a good idea. "throttling" the network connection can only lead to the node being expelled from the cluster.

    In order to see which path is taken for IO, local or NSD server, use 'mmlsdisk -M'.  Another way is to look at 'mmdiag --iohist' output.  The correct way to re-discover local disk path is indeed 'mmnsddiscover'.  If you run it and it doesn't help, something's wrong -- mmfsd can't see the disks, for whatever reason.  However, do verify that this is the case, using the commands above.

    yuri

  • perttu
    perttu
    3 Posts

    Re: How to direct IO from ethernet to FC?

    ‏2013-12-06T10:07:20Z  
    • yuri
    • ‏2013-12-06T02:59:58Z

    That's not a good idea. "throttling" the network connection can only lead to the node being expelled from the cluster.

    In order to see which path is taken for IO, local or NSD server, use 'mmlsdisk -M'.  Another way is to look at 'mmdiag --iohist' output.  The correct way to re-discover local disk path is indeed 'mmnsddiscover'.  If you run it and it doesn't help, something's wrong -- mmfsd can't see the disks, for whatever reason.  However, do verify that this is the case, using the commands above.

    yuri

    Hi yuri,

    "... way to re-discover ...". This may be the problem here. The local disks were not available at mount time.

    I had only a short period of time to set up these servers so I chose to start with nsd-disks only and then later add the local paths after I get the SAN configuration done. Evidently my presumption that I could later switch paths just like that was incorrect :(

    Anyway, thank You both very much for Your help.