Topic
  • 6 replies
  • Latest Post - ‏2012-12-18T20:33:47Z by FredStockatIBM
VincenzoVagnoni
VincenzoVagnoni
112 Posts

Pinned topic undue traffic on a client node

‏2012-12-15T11:50:13Z |
I have client nodes which are making relatively high GPFS traffic toward other client nodes (at about 50 MB/s), but without any process accessing GPFS. So this looks like internal GPFS message passing between client nodes. GPFS version is 3.4.0-17, on kernel 2.6.18-274.7.1.el5
Any reason for this? Can this be prevented somehow?
Updated on 2012-12-18T20:33:47Z at 2012-12-18T20:33:47Z by FredStockatIBM
  • VincenzoVagnoni
    VincenzoVagnoni
    112 Posts

    Re: undue traffic on a client node

    ‏2012-12-15T11:54:14Z  
    if it can help, you can find attached a full GPFS trace on one of such client nodes
  • VincenzoVagnoni
    VincenzoVagnoni
    112 Posts

    Re: undue traffic on a client node

    ‏2012-12-15T20:20:48Z  
    if it can help, you can find attached a full GPFS trace on one of such client nodes
    and this is a full dump on the node...
  • VincenzoVagnoni
    VincenzoVagnoni
    112 Posts

    Re: undue traffic on a client node

    ‏2012-12-15T20:22:36Z  
    and this is a full dump on the node...
    http://lhcbweb.bo.infn.it/dumpall.txt.gz
  • FredStockatIBM
    FredStockatIBM
    50 Posts

    Re: undue traffic on a client node

    ‏2012-12-15T23:43:07Z  
    In looking at the trace data it seems there are a good number of attribute and permission operations occurring in GPFS. Are you positive no process is walking through the file system and checking/adjusting extended attributes and file permissions?
  • VincenzoVagnoni
    VincenzoVagnoni
    112 Posts

    Re: undue traffic on a client node

    ‏2012-12-16T09:00:49Z  
    the node performs a series of metadata operations on demand from remote applications: basically sets some xattr and acl on directories and files, listing, creation/removal of directories, removal of files, etc.. But no I/O. The strange thing is that it started (after years of running smoothly) since few days to perform relatively large network I/O (see attached plot). The only difference in these last days is that we have upgraded client nodes in a remote cluster from 3.4.0-3 to 3.4.0-17. Might it be related?
  • FredStockatIBM
    FredStockatIBM
    50 Posts

    Re: undue traffic on a client node

    ‏2012-12-18T20:33:47Z  
    While you do not do any IO the changes to ACLs, file/dir creates, and attribute changes will cause GPFS to log those changes and flush them to storage. I honestly do not know why an upgrade would cause any change in traffic. Perhaps during the upgrade the cluster manager role for GPFS migrated to this node and that as resulted in the increased traffic. You can check where the cluster manager resides with "mmlsmgr -c".