Topic
  • 4 replies
  • Latest Post - ‏2013-03-28T09:44:27Z by SystemAdmin
VincenzoVagnoni
VincenzoVagnoni
112 Posts

Pinned topic problem with multi-filesytem cluster

‏2013-03-25T17:25:51Z |
we have a cluster with several filesystems, served by the same set of NSD servers. We are observing the following problem. When the disk backend of one of the filesystems saturates, we start observing large NSD thread I/O waiters on the NSD servers, and this is expected. Unfortunately, this ends up to consume the total number of NSD threads (nsdMaxWorkerThreads 256), and so all the operations of the other filesystems are delayed as well, since all NSD I/O threads are busy. Is there any smart way to handle this, apart from the obvious solution of using separate NSD's for different filesystems?
Updated on 2013-03-28T09:44:27Z at 2013-03-28T09:44:27Z by SystemAdmin
  • bdherr
    bdherr
    21 Posts

    Re: problem with multi-filesytem cluster

    ‏2013-03-26T14:53:13Z  
    What version of GPFS are you running? You could throttle down the clients by lowering the maxMBpS or lowering prefetchThreads, BUT this all depends on your workload and I/O access paterns and may have other unintended consequences.
  • VincenzoVagnoni
    VincenzoVagnoni
    112 Posts

    Re: problem with multi-filesytem cluster

    ‏2013-03-27T10:21:35Z  
    • bdherr
    • ‏2013-03-26T14:53:13Z
    What version of GPFS are you running? You could throttle down the clients by lowering the maxMBpS or lowering prefetchThreads, BUT this all depends on your workload and I/O access paterns and may have other unintended consequences.
    the GPFS version is 3.4.0-17. Lowering maxMBps or prefetch threads cannot be specified at filesystem level as far as I know, how could it help? We have thousands of computing clients and tens of different filesystems mounted on them. I don't think that what you suggest can be of any use. If there is no way to limit the number of NSD I/O threads per filesystem on a NSD server, it means that if a filesystem is saturating the backend storage, this has a huge impact on all other filesystems served by the same NSD server. I.e., one should avoid using the same NSD servers for different filesystems. But I have never read this in the GPFS documentation, nor in best practices and tutorials. Too bad...
  • bdherr
    bdherr
    21 Posts

    Re: problem with multi-filesytem cluster

    ‏2013-03-27T12:14:34Z  
    the GPFS version is 3.4.0-17. Lowering maxMBps or prefetch threads cannot be specified at filesystem level as far as I know, how could it help? We have thousands of computing clients and tens of different filesystems mounted on them. I don't think that what you suggest can be of any use. If there is no way to limit the number of NSD I/O threads per filesystem on a NSD server, it means that if a filesystem is saturating the backend storage, this has a huge impact on all other filesystems served by the same NSD server. I.e., one should avoid using the same NSD servers for different filesystems. But I have never read this in the GPFS documentation, nor in best practices and tutorials. Too bad...
    You are correct. There is no way to limit threads on a per filesystem basis.
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: problem with multi-filesytem cluster

    ‏2013-03-28T09:44:27Z  
    • bdherr
    • ‏2013-03-27T12:14:34Z
    You are correct. There is no way to limit threads on a per filesystem basis.
    Sure. Dont forget something.
    A clustered file system is a file system which is shared by being simultaneously mounted on multiple servers. There are several approaches to clustering, most of which do not employ a clustered file system (only direct attached storage for each node). Clustered file systems can provide features like location-independent addressing and redundancy which improve reliability or reduce the complexity of the other parts of the cluster. Parallel file systems are a type of clustered file system that spread data across multiple storage nodes, usually for redundancy or performance.