Topic
  • 6 replies
  • Latest Post - ‏2013-09-27T17:07:14Z by dlmcnabb
SystemAdmin
SystemAdmin
2092 Posts

Pinned topic Can gpfs inode be increased online (without re-mounting?)

‏2010-07-21T03:21:47Z |
Can gpfs inode be increased online (without re-mounting?)
Updated on 2010-07-22T09:23:49Z at 2010-07-22T09:23:49Z by SystemAdmin
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Can gpfs inode be increased online (without re-mounting?)

    ‏2010-07-21T04:07:55Z  
    {code}
    mmchfs <fsname> -F max#Inodes
    (code)
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: Can gpfs inode be increased online (without re-mounting?)

    ‏2010-07-21T09:29:03Z  
    • dlmcnabb
    • ‏2010-07-21T04:07:55Z
    {code}
    mmchfs <fsname> -F max#Inodes
    (code)
    Thanks for the command

    Can you give me a example for below formula

    maximum number of files = (total file system space/2) / (inode size + subblock size)

    1. total file system space = is this from df -k?
    2. inode size = is this from mmlsfs?
    3. subblock size = How to get subblock size?
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Can gpfs inode be increased online (without re-mounting?)

    ‏2010-07-21T20:20:44Z  
    Thanks for the command

    Can you give me a example for below formula

    maximum number of files = (total file system space/2) / (inode size + subblock size)

    1. total file system space = is this from df -k?
    2. inode size = is this from mmlsfs?
    3. subblock size = How to get subblock size?
    The "divide by 2" part of that test internally has been removed quite some time ago. Basically you need enough metadata space for all the inodes, and you need enough space for at lease one subblock per file (assuming it is pretty rare to have a lot of 0 length files). This formula does not take into account separate data and metadata disks nor the space for directories, but is still generous enough in most cases.

    The inode size is almost always 512 bytes: mmlsfs $fsname -i
    One subblock is always 1/32 of a fullblock: mmlsfs $fsname -B, or redundantly the fragment size: mmlsfs $fsname -f.

    You should consider what your average file size and number of entries per directory might be.

    But you can always add more disks and increase the number of inodes later.
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: Can gpfs inode be increased online (without re-mounting?)

    ‏2010-07-22T09:23:49Z  
    • dlmcnabb
    • ‏2010-07-21T20:20:44Z
    The "divide by 2" part of that test internally has been removed quite some time ago. Basically you need enough metadata space for all the inodes, and you need enough space for at lease one subblock per file (assuming it is pretty rare to have a lot of 0 length files). This formula does not take into account separate data and metadata disks nor the space for directories, but is still generous enough in most cases.

    The inode size is almost always 512 bytes: mmlsfs $fsname -i
    One subblock is always 1/32 of a fullblock: mmlsfs $fsname -B, or redundantly the fragment size: mmlsfs $fsname -f.

    You should consider what your average file size and number of entries per directory might be.

    But you can always add more disks and increase the number of inodes later.
    Thanks for detailed explanation
  • sabujp
    sabujp
    12 Posts

    Re: Can gpfs inode be increased online (without re-mounting?)

    ‏2013-09-27T12:33:47Z  
    • dlmcnabb
    • ‏2010-07-21T20:20:44Z
    The "divide by 2" part of that test internally has been removed quite some time ago. Basically you need enough metadata space for all the inodes, and you need enough space for at lease one subblock per file (assuming it is pretty rare to have a lot of 0 length files). This formula does not take into account separate data and metadata disks nor the space for directories, but is still generous enough in most cases.

    The inode size is almost always 512 bytes: mmlsfs $fsname -i
    One subblock is always 1/32 of a fullblock: mmlsfs $fsname -B, or redundantly the fragment size: mmlsfs $fsname -f.

    You should consider what your average file size and number of entries per directory might be.

    But you can always add more disks and increase the number of inodes later.

    mr mcnabb (or anyone else that knows for sure), 

    You mention "you need enough space for at least one subblock per file", so I'm assuming that means that I can only store 1 inode per metadata subblock and the rest of the subblock is used to store the data contents of the file should it fit in (subblock - inode) bytes? Does this occur for dataAndMetadata pools only or also for metadata only pools?

    I was confused since I ran "mmfsadm dump fs | grep -i inodes" and saw in this section :

     

    bgd_NeedFullScan 0 bgd_FullScanStarted 0 bgd_PreviousInodeScanIno -1

        inodes: size 512 (1 sectors), inodes per block 512, disk addrs 32

     

    Since my metadata block size is 256K, the above says i can get 512 inodes in a block. So which is true, 1 inode per subblock or as many inodes will fit into the blocksize depending on what I've set inode size to?

    Also, is there any performance benefit to increasing the inode size from 512 bytes? With my testing on RAID1 configured SSD's, I saw no performance difference with metadata operations, e.g. rm's, ls's and small file creation, using various metadata block sizes (16k - 16M) and varying inode sizes (512, 1024, or 4096) within those metadata block sizes.

    Thanks

     

     

     

    Updated on 2013-09-27T12:35:21Z at 2013-09-27T12:35:21Z by sabujp
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Can gpfs inode be increased online (without re-mounting?)

    ‏2013-09-27T17:07:14Z  
    • sabujp
    • ‏2013-09-27T12:33:47Z

    mr mcnabb (or anyone else that knows for sure), 

    You mention "you need enough space for at least one subblock per file", so I'm assuming that means that I can only store 1 inode per metadata subblock and the rest of the subblock is used to store the data contents of the file should it fit in (subblock - inode) bytes? Does this occur for dataAndMetadata pools only or also for metadata only pools?

    I was confused since I ran "mmfsadm dump fs | grep -i inodes" and saw in this section :

     

    bgd_NeedFullScan 0 bgd_FullScanStarted 0 bgd_PreviousInodeScanIno -1

        inodes: size 512 (1 sectors), inodes per block 512, disk addrs 32

     

    Since my metadata block size is 256K, the above says i can get 512 inodes in a block. So which is true, 1 inode per subblock or as many inodes will fit into the blocksize depending on what I've set inode size to?

    Also, is there any performance benefit to increasing the inode size from 512 bytes? With my testing on RAID1 configured SSD's, I saw no performance difference with metadata operations, e.g. rm's, ls's and small file creation, using various metadata block sizes (16k - 16M) and varying inode sizes (512, 1024, or 4096) within those metadata block sizes.

    Thanks

     

     

     

    The "one inode + one subblock" is just an extremely rough estimate of the maximum number of files that would fit on your existing disks. It does not say anything about how those are allocated. The "inode 0" file in GPFS is the file of all the inodes. It is allocated in blocksize chunks and each block is filled with inodes of whatever the configured inode size is. Then the calculation assumes that if you created a small file using each inode, they might each use at least one subblock. It is just trying t determin a theoretical limit on the number of inodes.

    The calculation does not even look at whether there are separate data and metadata disks. Disk usage can be changed using mmchdisk.

    As of release 3.5 this calculation is even more vague, since GPFS will store file/directory/symlink data inside the inode itself if the file is small enough. It may not even need a subblock. Most small directories will fall into this category. An inode has a small fixed header of roughly 128 bytes. The rest of the inode can be used for a mixture of disk block addresses, Extended Attributes, and/or data.

    You can create filesystem with larger inodes, and thus store more small files directly in the inode. This can make reading these small files much more efficient since the data read is free after reading the inode.