Topic
6 replies Latest Post - ‏2012-12-18T20:33:47Z by FredStockatIBM
VincenzoVagnoni
VincenzoVagnoni
108 Posts
ACCEPTED ANSWER

Pinned topic undue traffic on a client node

‏2012-12-15T11:50:13Z |
I have client nodes which are making relatively high GPFS traffic toward other client nodes (at about 50 MB/s), but without any process accessing GPFS. So this looks like internal GPFS message passing between client nodes. GPFS version is 3.4.0-17, on kernel 2.6.18-274.7.1.el5
Any reason for this? Can this be prevented somehow?
Updated on 2012-12-18T20:33:47Z at 2012-12-18T20:33:47Z by FredStockatIBM
  • VincenzoVagnoni
    VincenzoVagnoni
    108 Posts
    ACCEPTED ANSWER

    Re: undue traffic on a client node

    ‏2012-12-15T11:54:14Z  in response to VincenzoVagnoni
    if it can help, you can find attached a full GPFS trace on one of such client nodes
  • FredStockatIBM
    FredStockatIBM
    43 Posts
    ACCEPTED ANSWER

    Re: undue traffic on a client node

    ‏2012-12-15T23:43:07Z  in response to VincenzoVagnoni
    In looking at the trace data it seems there are a good number of attribute and permission operations occurring in GPFS. Are you positive no process is walking through the file system and checking/adjusting extended attributes and file permissions?
  • VincenzoVagnoni
    VincenzoVagnoni
    108 Posts
    ACCEPTED ANSWER

    Re: undue traffic on a client node

    ‏2012-12-16T09:00:49Z  in response to VincenzoVagnoni
    the node performs a series of metadata operations on demand from remote applications: basically sets some xattr and acl on directories and files, listing, creation/removal of directories, removal of files, etc.. But no I/O. The strange thing is that it started (after years of running smoothly) since few days to perform relatively large network I/O (see attached plot). The only difference in these last days is that we have upgraded client nodes in a remote cluster from 3.4.0-3 to 3.4.0-17. Might it be related?
  • FredStockatIBM
    FredStockatIBM
    43 Posts
    ACCEPTED ANSWER

    Re: undue traffic on a client node

    ‏2012-12-18T20:33:47Z  in response to VincenzoVagnoni
    While you do not do any IO the changes to ACLs, file/dir creates, and attribute changes will cause GPFS to log those changes and flush them to storage. I honestly do not know why an upgrade would cause any change in traffic. Perhaps during the upgrade the cluster manager role for GPFS migrated to this node and that as resulted in the increased traffic. You can check where the cluster manager resides with "mmlsmgr -c".