Topic
  • 7 replies
  • Latest Post - ‏2014-03-29T19:01:07Z by chr78
Itkovian
Itkovian
2 Posts

Pinned topic negative GPFS quota reported on dependent filesets

‏2013-09-04T09:13:35Z | filesets quota

Hi,

We have a GPFS setup where we have (amongst others) a root fileset and several filesets for user groups. We come from a 3.2 setup, where we did not have the possibility to use independent filesets with their own inode space, so everything ultimately derives from the root fileset, inode-wise. At least, that is my understanding of the setup. We have recently updated to 3.5.0.11, but the inode spaces obvisouly did not turn magically into independent sets.

What we now see is the following when running mmrepquota (or mmlsquota):

user1   root       USR       -25422944 4930928640 5190451200     163840     none |      -93       0        0       93     none
user1   g1   USR       498963808 4930928640 5190451200          0     none |   535333       0        0        0     none
user1   g1   GRP       511455424          0          0          0     none |   628607       0        0        0     none 

So, the softquota is broken on g1 but no grace is shown. And the quota on the root fileset is reported as being negative. We ran mmcheckquota, which resolved things, yet 30 minutes later, we again see negative quota being reported. 

Any suggestions on how to fix/avoid this? Is this related to upgrading to 3.5.x, and keeping dependent filesets? We do not see this behaviou on e.g., 3.5.x with independent filesets.

Thanks in advance,

-- Andy 

  • chr78
    chr78
    144 Posts

    Re: negative GPFS quota reported on dependent filesets

    ‏2013-09-04T11:32:42Z  

    Not sure whether this is related or not, but we have seen similar issues with 3.5.0.11 && default-perfilesetquotas

    on dependent filesets. We did not have negative quota values reported but totally wrong over-quota situations.

    We had to regularily run mmcheckquota, as well.

    This issue was definetely not seen when we were at 3.5.0.9/3.5.0.10 levels...

    cheers.

  • chr78
    chr78
    144 Posts

    Re: negative GPFS quota reported on dependent filesets

    ‏2013-09-05T11:52:39Z  
    • chr78
    • ‏2013-09-04T11:32:42Z

    Not sure whether this is related or not, but we have seen similar issues with 3.5.0.11 && default-perfilesetquotas

    on dependent filesets. We did not have negative quota values reported but totally wrong over-quota situations.

    We had to regularily run mmcheckquota, as well.

    This issue was definetely not seen when we were at 3.5.0.9/3.5.0.10 levels...

    cheers.

    actually, we do see the negative values, as well. (I just did not use mmrepquota before but simply mmlsquota). Now the issue

    is seen also with 3.5.0.12.

  • Itkovian
    Itkovian
    2 Posts

    Re: negative GPFS quota reported on dependent filesets

    ‏2013-09-06T22:31:52Z  

    Some more information.

    This is the state of the user quota and of the fileset where the user has rights as well:

     

     

    user1   root       USR        -5972096          0          0     871776     none |    -6718       0        0      896     none
    user1   g1   USR       538133280 4930928640 5190451200     138112     none |   549942       0        0       83     none
    user1   g1   GRP       650738592          0          0          0     none |   643216       0        0        0     none
    g1   root       FILESET  5351015008 9861857280 10380902400     158624     none |  3047208       0        0       74     none

    After this I wrote 100 GiB to this users storage space under the g1 fileset. This is what we see then:

     

    user1   root       USR        -5972096          0          0     871776     none |    -6718       0        0      896     none
    user1   g1   USR       638237472 4930928640 5190451200          0     none |   549940       0        0        0     none
    user1   g1   GRP       650738592          0          0          0     none |   643216       0        0        0     none
    g1   root       FILESET  5351015008 9861857280 10380902400     158624     none |  3047208       0        0       74     none

    So, essentially, there is no change is usage in the root fileset, yet there is an increase in both USR and FILESET quota for the user1 under g1 and g1, respectively.

    However, when I delete the 100 GiB file, I see this (about 15 minutes later)

    user1   root       USR      -105931904          0          0    1035616     none |    -6718       0        0      896     none
    user1   g1   USR       638214400 4930928640 5190451200      63168     none |   549945       0        0       37     none
    user1   g1   GRP       650715520          0          0      63168     none |   643221       0        0       37     none
    g1   root       FILESET  5251032384 9861857280 10380902400     365760     none |  3047213       0        0       89     none
     

    Thus, the USR quota for user1 under g1 has not been adjusted, but the user1 USR quota for the root fileset has decreased further, with 100 GiB. However, we do see that the FILESET quota for g1 is back to its previous level, so that's OK.

    I have tried the same thing with a different user and there I do _not_ see this happening.

    Regards,

    -- Andy

     

  • HajoEhlers
    HajoEhlers
    254 Posts

    Re: negative GPFS quota reported on dependent filesets

    ‏2013-11-19T10:33:09Z  

    On GPFS 3.4.0.23  we see the same problem ...

    mmrepquota: 

    ...

    MMDS       FILESET  -1205008640

    ...

    starting after changing the replication settings 

    -m DefaultMetadataReplicas
    -r DefaultDataReplicas]

    via mmchfs from 1 to 2 but having all disk on 1 failure group. A mmcheckquota fixed the quota . In case the we see the problem again i will reset the value for -m , -r back to 1.

    Note

    We see negative quotas on filesets and to low values for user quotas. ( different GPFS )

    cheers

    Hajo

     

    Updated on 2013-11-19T10:34:50Z at 2013-11-19T10:34:50Z by HajoEhlers
  • HajoEhlers
    HajoEhlers
    254 Posts

    Re: negative GPFS quota reported on dependent filesets

    ‏2013-11-19T16:26:09Z  

    On GPFS 3.4.0.23  we see the same problem ...

    mmrepquota: 

    ...

    MMDS       FILESET  -1205008640

    ...

    starting after changing the replication settings 

    -m DefaultMetadataReplicas
    -r DefaultDataReplicas]

    via mmchfs from 1 to 2 but having all disk on 1 failure group. A mmcheckquota fixed the quota . In case the we see the problem again i will reset the value for -m , -r back to 1.

    Note

    We see negative quotas on filesets and to low values for user quotas. ( different GPFS )

    cheers

    Hajo

     

    FYI:

    Even after running mmcheckquota the problem with the negative quotas showed up within minutes. So i reset the 

    -m DefaultMetadataReplicas
    -r DefaultDataReplicas

     

    back to one(1), run an mmcheckquota again and the problem did not showed up so far.

     

    Might be related to the fact that in case the DefaultMetadataReplicas and DefaultDataReplicas are changed all file are illplaced/unbalanced.

    At least it looks like that the problem is easily to reproduce.

     

  • yuri
    yuri
    305 Posts

    Re: negative GPFS quota reported on dependent filesets

    ‏2013-11-23T06:41:51Z  

    FYI:

    Even after running mmcheckquota the problem with the negative quotas showed up within minutes. So i reset the 

    -m DefaultMetadataReplicas
    -r DefaultDataReplicas

     

    back to one(1), run an mmcheckquota again and the problem did not showed up so far.

     

    Might be related to the fact that in case the DefaultMetadataReplicas and DefaultDataReplicas are changed all file are illplaced/unbalanced.

    At least it looks like that the problem is easily to reproduce.

     

    I'd recommend opening a PMR for this.  The code clearly isn't working correctly.

    yuri

  • chr78
    chr78
    144 Posts

    Re: negative GPFS quota reported on dependent filesets

    ‏2014-03-29T19:01:07Z  
    • yuri
    • ‏2013-11-23T06:41:51Z

    I'd recommend opening a PMR for this.  The code clearly isn't working correctly.

    yuri

    it seems as this issue was fixed in PTF 16 or 17 - at least I do not see negative quotas

    on a problematic system that is running PTF 17 since about two weeks. Could someone

    confirm this? cheers.