Topic
  • 3 replies
  • Latest Post - ‏2012-09-08T09:29:56Z by ufa
ufa
ufa
133 Posts

Pinned topic Remote cluster vs. Local Cluster performance- advantages and disadvantages

‏2012-09-06T12:22:50Z |
Hi,
I got a general question:
are there any fundamental differences in client performance and system stability between
having a large number of client nodes (a few thousand) together with the NSD servers in one big GPFS cluster OTOH and having the clients in several smaller clusters mounting the file systems remotely from one storage cluster comprisiung the NSD servers?

In particular I was told by someone that the "all-in-one" solution would yield a much better performance in HPC environments than using remotely mounting client clusters. However, the explanation why remained vague.

Is there a general difference in accessing data and metadata from within a cluster and from another, remotely mounting one? What are the benefits of these approaches vs the respective other one?
I could imagine that within a cluster one has all-to-all connections right from the start, whereas connections to remote cluster nodes will be opened as required only.

Thank you in advance for your efforts
Uwe
Updated on 2012-09-08T09:29:56Z at 2012-09-08T09:29:56Z by ufa
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: Remote cluster vs. Local Cluster performance- advantages and disadvantages

    ‏2012-09-06T19:25:07Z  
    Hi Uwe,

    I can not vouch for the GPFS internals or any particular performance. But the standard setup in a Bluegene supercomputer environment is with dedicated GPFS clusters for the storage (NSDs) and a separate cluster for the GPFS clients (Bluegene IO-nodes). At least in terms on bandwidth there is no disadvantage to this. We have done some tuning work and we able to achieve application performance very close to the hardware/wire speed.

    If there is an advantage for you to split your GPFS in several independent clusters, then I would not hesitate to do that.

    Markus
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: Remote cluster vs. Local Cluster performance- advantages and disadvantages

    ‏2012-09-06T21:05:07Z  
    There's no substantial difference in regular IO performance for remote vs local cluster nodes when only two clusters are involved. When two (or more) remote clusters access the same subset of files, things get more complicated, because GPFS has an obligation to avoid direct network communication between nodes in different remote clusters, but this still only has a minimal impact on IO performance. Once a remote node joins the home cluster, by the virtue of mounting a remote file system, it's treated like a regular node for normal IO purposes. What is different for remote clusters is administration -- there are separate clusters that need to be managed independently. Also, the set of nodes to use as workers for distributed admin commands, e.g. mmrestripefs, is restricted to local cluster nodes.

    yuri
  • ufa
    ufa
    133 Posts

    Re: Remote cluster vs. Local Cluster performance- advantages and disadvantages

    ‏2012-09-08T09:29:56Z  
    Thank you for your replies. That supports what we plan to do, however, if somebody argues bluntly he has seen contradicting scenarios one should at least ask, so forgive my ignorance.