Topic
4 replies Latest Post - ‏2013-12-06T10:07:20Z by perttu
perttu
perttu
3 Posts
ACCEPTED ANSWER

Pinned topic How to direct IO from ethernet to FC?

‏2013-12-03T15:15:49Z |

Hi all,

I have configured both ethernet access (throw nsd-servers) and fibre channel access (npiv) to my GPFS -disks. When I mount the filesystems IO-traffic goes through  disks and HBA:s if the disks are available.  If the disks are not available the filesystems get mounted through nds-servers and IO-traffic goes through en-devices. Unfortunately I haven't find a non-disruptive way to redirect IO-traffic to go through disks instead of ethernet.

How do I change the traffic routing from ethernet to disk without unmounting and mounting?

  • dlmcnabb
    dlmcnabb
    1012 Posts
    ACCEPTED ANSWER

    Re: How to direct IO from ethernet to FC?

    ‏2013-12-03T16:19:31Z  in response to perttu

    If the disks become available after being missing, try running

    mmnsddiscover -a -N all

    This will first have all NSD servers check for devices coming back, and then all the client nodes will reevaluate whether they see disks locally or served by NSD servers.

     

    If devices are not visible on nodes fast enough when GPFS first comes up, then you may have to put some code in /var/mmfs/etc/gpfsready script to check that disks are available before letting GPFS start any mounts. This script blocks GPFS starting until it returns.

    • perttu
      perttu
      3 Posts
      ACCEPTED ANSWER

      Re: How to direct IO from ethernet to FC?

      ‏2013-12-05T14:11:07Z  in response to dlmcnabb

      Thank You for Your help.

      Unfortunately mmnsddiscover didn't do it :( mmlsnsd -m shows that disk
      is available locally but when I start writing to disk network traffic
      peeks and disk io stays flat.

      However, from what I've read change should be automatic:

      - mount -option useNSDserver=asneeded is default

      - failover shoud happen from local disk to nsd and vice verca

      Maybe I should try to throttle network and force failover that way?

      Updated on 2013-12-05T14:55:32Z at 2013-12-05T14:55:32Z by perttu
      • yuri
        yuri
        185 Posts
        ACCEPTED ANSWER

        Re: How to direct IO from ethernet to FC?

        ‏2013-12-06T02:59:58Z  in response to perttu

        That's not a good idea. "throttling" the network connection can only lead to the node being expelled from the cluster.

        In order to see which path is taken for IO, local or NSD server, use 'mmlsdisk -M'.  Another way is to look at 'mmdiag --iohist' output.  The correct way to re-discover local disk path is indeed 'mmnsddiscover'.  If you run it and it doesn't help, something's wrong -- mmfsd can't see the disks, for whatever reason.  However, do verify that this is the case, using the commands above.

        yuri

        • perttu
          perttu
          3 Posts
          ACCEPTED ANSWER

          Re: How to direct IO from ethernet to FC?

          ‏2013-12-06T10:07:20Z  in response to yuri

          Hi yuri,

          "... way to re-discover ...". This may be the problem here. The local disks were not available at mount time.

          I had only a short period of time to set up these servers so I chose to start with nsd-disks only and then later add the local paths after I get the SAN configuration done. Evidently my presumption that I could later switch paths just like that was incorrect :(

          Anyway, thank You both very much for Your help.