Topic
  • 10 replies
  • Latest Post - ‏2013-09-13T21:07:20Z by dlmcnabb
GPFSnerd
GPFSnerd
10 Posts

Pinned topic Calculate GPFS memory consumption on AIX

‏2013-08-02T15:33:36Z |

Hi everybody,

 

i would like to know how i can find out the real GPFS memory consumption on AIX. When i launch mmdiag --memory or svmon, i don't  know how to use the output and the output doesn't show the same amount :

(test) [root] /home/root > mmdiag --memory
 
=== mmdiag: memory ===
mmfsd heap size: 37949952 bytes
 
 
Statistics for MemoryPool id 1 ("Shared Segment (EPHEMERAL)")
         128 bytes in use
  1019058540 hard limit on memory usage
      262144 bytes committed to regions
          66 allocations
          66 frees
           0 allocation failures
 
 
Statistics for MemoryPool id 2 ("Shared Segment")
    47449912 bytes in use
  1019058540 hard limit on memory usage
    54657024 bytes committed to regions
    13099047 allocations
    13092864 frees
           0 allocation failures
 
 
Statistics for MemoryPool id 3 ("Token Manager")
    19281232 bytes in use
   510027355 hard limit on memory usage
    21364736 bytes committed to regions
        3973 allocations
        2823 frees
           0 allocation failures
 
(test) [root] /home/root > svmon -P 11927758
 
-------------------------------------------------------------------------------
     Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
11927758 mmfsd64         221049   196809        0   215846      Y     Y     Y
 
    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
  831ec3  a0000001 work N/A                          s  65536 65536    0   65536
   20002         0 work kernel segment               L     16    16    0      16
  a41e64  a0000000 work N/A                          s  65536 65536    0   65536
  990019  90000000 work shared library text          s   8797     0    0    8797
  9f0cdf        11 work text data BSS heap           s   5432     0    0    5432
  bd0c7d        10 clnt text data BSS heap,          s   5140     0    -       -
                        /dev/bos_hd2:172035
   50005  9ffffffd work shared library               s   2541     0    0    2541
  9b001b  90020014 work shared library               s   1141     0    0    1141
  901e10         - work                              s    906   179    0     906
  a71ea7  9001000a work shared library data          s    179     0    0     179
  881e88  80020014 work USLA heap                    s    165     0    0     165
  110011  9ffffffe work shared library               s     36     0    0      36
  be1ebe         - clnt /dev/bos_hd2:176132          s     34     0    -       -
  811e01 f00000002 work process private              s     32    22    0      32
  8200c2  9fffffff clnt USLA text,/dev/bos_hd2:4158  s     20     0    -       -
  ac1eac         - clnt /dev/bos_hd9var:159          s      9     0    -       -
  871e07  8fffffff work private load data            s      5     0    0       5
  961d96  ffffffff work application stack            s      4     0    0       4
  9d1e9d  fffffff0 work application stack            s      0     0    0       0
  9c1ddc  fffffff9 work application stack            s      0     0    0       0
  ae1eae  fffffff3 work application stack            s      0     0    0       0
  b01e30  fffffff4 work application stack            s      0     0    0       0
  b21e32  fffffff5 work application stack            s      0     0    0       0
  8b1e4b  fffffffd work application stack            s      0     0    0       0
  8a1e4a  fffffff1 work application stack            s      0     0    0       0
  891e89  fffffff6 work application stack            s      0     0    0       0
  b41e34  fffffffc work application stack            s      0     0    0       0
  ba1eba  fffffff7 work application stack            s      0     0    0       0
  841e84  fffffffe work application stack            s      0     0    0       0
  a21ea2  fffffff2 work application stack            s      0     0    0       0
  be193e  fffffff8 work application stack            s      0     0    0       0
  801e40  fffffffa work application stack            s      0     0    0       0
  9d1e1d  fffffffb work application stack            s      0     0    0       0
  ac1dec  70000400 mmap maps 1 source(s)             s      0     0    -       -
 

The shared segment is not visible with ipcs (normal ???) :

 

(test) [root] /home/root > ipcs -bm

IPC status from /dev/mem as of Fri Aug  2 15:29:33 CUT 2013
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   6291457 0x6c000022 --rw-------   zabbix   zabbix  15056312
m   2097154 0x61003008 --rw------- pconsole   system  10485760
 

Does someone know how to calculate the real GPFS memory consumption, the maximum amount of memory that GPFS can use (i don't mean "pagepoolMaxPhysMemPct" parameter) and the amount of memory pinned vs unpinned ? Thank you

 

  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-08-02T16:02:47Z  

    The main pinned memory is the pagepool which I think svmon shows as the two segments of 65536 pages. GPFS also pins a small fixed size header of the "Shared Segment" pool.

    The heap, most of the "Shared Segment", and all of the "Token Manager" pools are not pinned and only allocated as needed. the "mmdiag memory dump shows how much is allocated here. I have no idea how that shows up in svmon.

    Other memory that used in the kernel for AIX vnode and gnode structures is proportional to the maxFilesToCache+maxStatCache GPFS settings. These are pinned when created and only as needed. Svmon does not show this usage.

  • GPFSnerd
    GPFSnerd
    10 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-08-05T08:36:45Z  
    • dlmcnabb
    • ‏2013-08-02T16:02:47Z

    The main pinned memory is the pagepool which I think svmon shows as the two segments of 65536 pages. GPFS also pins a small fixed size header of the "Shared Segment" pool.

    The heap, most of the "Shared Segment", and all of the "Token Manager" pools are not pinned and only allocated as needed. the "mmdiag memory dump shows how much is allocated here. I have no idea how that shows up in svmon.

    Other memory that used in the kernel for AIX vnode and gnode structures is proportional to the maxFilesToCache+maxStatCache GPFS settings. These are pinned when created and only as needed. Svmon does not show this usage.

    The main pinned memory is the pagepool which I think svmon shows as the two segments of 65536 pages. GPFS also pins a small fixed size header of the "Shared Segment" pool.

    The heap, most of the "Shared Segment", and all of the "Token Manager" pools are not pinned and only allocated as needed. the "mmdiag memory dump shows how much is allocated here. I have no idea how that shows up in svmon.

    Other memory that used in the kernel for AIX vnode and gnode structures is proportional to the maxFilesToCache+maxStatCache GPFS settings. These are pinned when created and only as needed. Svmon does not show this usage.

    Regarding the Pagepool in SVMON, you're right it matches with my clusters. What do you mean by "small fixed size header of the "Shared Segment" pool" ? What is it for ?

    You said that most of the Shared Segment is not pinned (The heap) but i thought that the Shared Segment (represented by memory pool id 1 & 2 in mmdiag --memory but i may be wrong) was composed of the Pagepool (pinned memory) & a small non-pinned area (page 34/35 of Concepts, Planning & Installation guide) including a full inode cache (maxFilesToCache) and a stat cache (maxStatCache). Can you clarify this point please ?

    What is "hard limit on memory usage" in mmdiag --memory command ? Does it mean that my server can only get a 2GB shared segment (1019058540 hard limit on memory usage for memory pool id 1 and  2) ?

    Last question : Is it normal not to see the Shared Segment in IPCS -m command ?

     

    Thank you for your help

     

    Updated on 2013-08-05T09:10:17Z at 2013-08-05T09:10:17Z by GPFSnerd
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-08-05T17:09:41Z  
    • GPFSnerd
    • ‏2013-08-05T08:36:45Z

    The main pinned memory is the pagepool which I think svmon shows as the two segments of 65536 pages. GPFS also pins a small fixed size header of the "Shared Segment" pool.

    The heap, most of the "Shared Segment", and all of the "Token Manager" pools are not pinned and only allocated as needed. the "mmdiag memory dump shows how much is allocated here. I have no idea how that shows up in svmon.

    Other memory that used in the kernel for AIX vnode and gnode structures is proportional to the maxFilesToCache+maxStatCache GPFS settings. These are pinned when created and only as needed. Svmon does not show this usage.

    Regarding the Pagepool in SVMON, you're right it matches with my clusters. What do you mean by "small fixed size header of the "Shared Segment" pool" ? What is it for ?

    You said that most of the Shared Segment is not pinned (The heap) but i thought that the Shared Segment (represented by memory pool id 1 & 2 in mmdiag --memory but i may be wrong) was composed of the Pagepool (pinned memory) & a small non-pinned area (page 34/35 of Concepts, Planning & Installation guide) including a full inode cache (maxFilesToCache) and a stat cache (maxStatCache). Can you clarify this point please ?

    What is "hard limit on memory usage" in mmdiag --memory command ? Does it mean that my server can only get a 2GB shared segment (1019058540 hard limit on memory usage for memory pool id 1 and  2) ?

    Last question : Is it normal not to see the Shared Segment in IPCS -m command ?

     

    Thank you for your help

     

    The "Shared Segment" is a private memory segment allocated by special AIX kernel methods (not visible to ipcs). (Ignore memory pool 1. It is a transient pool of memory to avoid fragmenting the Shared Segment in memory pool 2) The Shared Segment is used to share metadata in the between the kernel space and the GPFS daemon. Applications making filesystem requests, go into the kernel where it can map the shared segment and look at GPFS metadata in order to see if there are things like buffers in the pagepool cache for the data they want to read or write. If it's not cached, then the kernel thread makes requests up to the GPFS daemon to make the data available. Only the header of the Shared Segment needs to be pinned for these requests up to the daemon. The rest of the segment is pageable. The amount it needs depends on the the settings of maxFilesToCache+maxStatCache and a few other settings. See "mmfsadm dump fs" for the calculations of how much memory is used based on these settings. This is internal debugging data for developers, so is not documented anywhere. But you may find something useful.

    The "hard limit" is just based on your configuration settings which can be much larger if desired.

  • GPFSnerd
    GPFSnerd
    10 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-08-05T19:51:43Z  
    • dlmcnabb
    • ‏2013-08-05T17:09:41Z

    The "Shared Segment" is a private memory segment allocated by special AIX kernel methods (not visible to ipcs). (Ignore memory pool 1. It is a transient pool of memory to avoid fragmenting the Shared Segment in memory pool 2) The Shared Segment is used to share metadata in the between the kernel space and the GPFS daemon. Applications making filesystem requests, go into the kernel where it can map the shared segment and look at GPFS metadata in order to see if there are things like buffers in the pagepool cache for the data they want to read or write. If it's not cached, then the kernel thread makes requests up to the GPFS daemon to make the data available. Only the header of the Shared Segment needs to be pinned for these requests up to the daemon. The rest of the segment is pageable. The amount it needs depends on the the settings of maxFilesToCache+maxStatCache and a few other settings. See "mmfsadm dump fs" for the calculations of how much memory is used based on these settings. This is internal debugging data for developers, so is not documented anywhere. But you may find something useful.

    The "hard limit" is just based on your configuration settings which can be much larger if desired.

    The "Shared Segment" is a private memory segment allocated by special AIX kernel methods (not visible to ipcs). (Ignore memory pool 1. It is a transient pool of memory to avoid fragmenting the Shared Segment in memory pool 2) The Shared Segment is used to share metadata in the between the kernel space and the GPFS daemon. Applications making filesystem requests, go into the kernel where it can map the shared segment and look at GPFS metadata in order to see if there are things like buffers in the pagepool cache for the data they want to read or write. If it's not cached, then the kernel thread makes requests up to the GPFS daemon to make the data available. Only the header of the Shared Segment needs to be pinned for these requests up to the daemon. The rest of the segment is pageable. The amount it needs depends on the the settings of maxFilesToCache+maxStatCache and a few other settings. See "mmfsadm dump fs" for the calculations of how much memory is used based on these settings. This is internal debugging data for developers, so is not documented anywhere. But you may find something useful.

    The "hard limit" is just based on your configuration settings which can be much larger if desired.

    Thank you for your reply, it´s very interesting to get some extra details on GPFS. I have still some questions :

    - How is it possible to know the pagepool consumption as it´s a pinned memory fully allocated at the GPFS daemon startup ? Is the pagepool represented by the memory pool id 2 ? If you think so how does it come that only the header is pinned (the pagepool can be huge) ?

    - Is it possible to see memory pool id 3 (token management) consumption with SVMON ?

    - How does it come that a memory pool id 2 hard limit is 1GB when the pagepool is set to 2GB ?

    - Do you know if a documentation on mmfsadm exists ?

     

    Updated on 2013-08-05T19:53:06Z at 2013-08-05T19:53:06Z by GPFSnerd
  • GPFSnerd
    GPFSnerd
    10 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-08-13T14:46:49Z  
    • GPFSnerd
    • ‏2013-08-05T19:51:43Z

    The "Shared Segment" is a private memory segment allocated by special AIX kernel methods (not visible to ipcs). (Ignore memory pool 1. It is a transient pool of memory to avoid fragmenting the Shared Segment in memory pool 2) The Shared Segment is used to share metadata in the between the kernel space and the GPFS daemon. Applications making filesystem requests, go into the kernel where it can map the shared segment and look at GPFS metadata in order to see if there are things like buffers in the pagepool cache for the data they want to read or write. If it's not cached, then the kernel thread makes requests up to the GPFS daemon to make the data available. Only the header of the Shared Segment needs to be pinned for these requests up to the daemon. The rest of the segment is pageable. The amount it needs depends on the the settings of maxFilesToCache+maxStatCache and a few other settings. See "mmfsadm dump fs" for the calculations of how much memory is used based on these settings. This is internal debugging data for developers, so is not documented anywhere. But you may find something useful.

    The "hard limit" is just based on your configuration settings which can be much larger if desired.

    Thank you for your reply, it´s very interesting to get some extra details on GPFS. I have still some questions :

    - How is it possible to know the pagepool consumption as it´s a pinned memory fully allocated at the GPFS daemon startup ? Is the pagepool represented by the memory pool id 2 ? If you think so how does it come that only the header is pinned (the pagepool can be huge) ?

    - Is it possible to see memory pool id 3 (token management) consumption with SVMON ?

    - How does it come that a memory pool id 2 hard limit is 1GB when the pagepool is set to 2GB ?

    - Do you know if a documentation on mmfsadm exists ?

     

    Any help ?

  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-08-27T15:18:29Z  
    • GPFSnerd
    • ‏2013-08-05T19:51:43Z

    The "Shared Segment" is a private memory segment allocated by special AIX kernel methods (not visible to ipcs). (Ignore memory pool 1. It is a transient pool of memory to avoid fragmenting the Shared Segment in memory pool 2) The Shared Segment is used to share metadata in the between the kernel space and the GPFS daemon. Applications making filesystem requests, go into the kernel where it can map the shared segment and look at GPFS metadata in order to see if there are things like buffers in the pagepool cache for the data they want to read or write. If it's not cached, then the kernel thread makes requests up to the GPFS daemon to make the data available. Only the header of the Shared Segment needs to be pinned for these requests up to the daemon. The rest of the segment is pageable. The amount it needs depends on the the settings of maxFilesToCache+maxStatCache and a few other settings. See "mmfsadm dump fs" for the calculations of how much memory is used based on these settings. This is internal debugging data for developers, so is not documented anywhere. But you may find something useful.

    The "hard limit" is just based on your configuration settings which can be much larger if desired.

    Thank you for your reply, it´s very interesting to get some extra details on GPFS. I have still some questions :

    - How is it possible to know the pagepool consumption as it´s a pinned memory fully allocated at the GPFS daemon startup ? Is the pagepool represented by the memory pool id 2 ? If you think so how does it come that only the header is pinned (the pagepool can be huge) ?

    - Is it possible to see memory pool id 3 (token management) consumption with SVMON ?

    - How does it come that a memory pool id 2 hard limit is 1GB when the pagepool is set to 2GB ?

    - Do you know if a documentation on mmfsadm exists ?

     

    The pagepool and the Shared Segment (memory pools 2 and 1) are two different entities. The Shared Segment is mostly used to cache metadata information about files and buffers. The pagepool is used to cache data and metadata buffers that come from disk or going to disk.

    The pagepool is always pinned to reduce pinning it every time an IO is done. The Shared Segment on AIX is only partially pinned for data structures that may be looked at by interrupt process when doing IO. However, on Linux the Shared Segment is allocated from the kernel vmalloc space which is always pinned.

    The Token Memory pool 3 should show up as one of the pageable segments in svmon. I cannot tell which one.

    There is no doc for mmfsadm. It is an internal debugging tool that changes all the time.

     

  • GPFSnerd
    GPFSnerd
    10 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-09-03T10:53:25Z  
    • dlmcnabb
    • ‏2013-08-27T15:18:29Z

    The pagepool and the Shared Segment (memory pools 2 and 1) are two different entities. The Shared Segment is mostly used to cache metadata information about files and buffers. The pagepool is used to cache data and metadata buffers that come from disk or going to disk.

    The pagepool is always pinned to reduce pinning it every time an IO is done. The Shared Segment on AIX is only partially pinned for data structures that may be looked at by interrupt process when doing IO. However, on Linux the Shared Segment is allocated from the kernel vmalloc space which is always pinned.

    The Token Memory pool 3 should show up as one of the pageable segments in svmon. I cannot tell which one.

    There is no doc for mmfsadm. It is an internal debugging tool that changes all the time.

     

    So do you have a calculation rule to calculate the amount of memory used by GPFS ?

  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-09-06T14:56:47Z  
    • GPFSnerd
    • ‏2013-09-03T10:53:25Z

    So do you have a calculation rule to calculate the amount of memory used by GPFS ?

    The beginning of the "mmfsadm dump fs" shows the calculations of how the Shared Segment is allocated based on the configuration settings.

    There is no rule of thumb for a cache since it is all very dependent on what the application load is expected or desired to be.

  • GPFSnerd
    GPFSnerd
    10 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-09-12T10:20:55Z  
    • dlmcnabb
    • ‏2013-09-06T14:56:47Z

    The beginning of the "mmfsadm dump fs" shows the calculations of how the Shared Segment is allocated based on the configuration settings.

    There is no rule of thumb for a cache since it is all very dependent on what the application load is expected or desired to be.

    I mean how can i calculate with AIX and GPFS commands the amount of memory used on the system for GPFS ?

    I need this information because we can't figure out and bench this part of GPFS. Is it Shared Segment consumption in "mmfsadm dump fs" + Pagepool visible in SVMON + heap size in "mmdiag --memory" + The Token Memory pool 3 ? (Is the "mmfsadm dump fs" shows the actual used memory or the theorical used memory with the umalloc defined limits ?)

    If i check mmfsadm dump fs, i get :

    Filesystem dump:
      UMALLOC limits:
        bufferDescLimit      10000 desired    10000
        fileCacheLimit        1000 desired     1000
        statCacheLimit        4000 desired     4000
        diskAddrBuffLimit      800 desired      800
     
        buffDescMem          8047 k  =   10000 *  824 bytes
        fileCacheMem       878223 k  =  280330 * 3208 bytes (inode size 512 + 2696)
        statCacheMem         1281 k  =    4000 *  328 bytes
        diskAddrBufMem        113 k  =     800 *  144 bytes
        allocSegmentMem      2500 k  =   10000 *  256 bytes
        otherMem           157389 k
        mdTemp mem            256 k
     

    With mmdiag --memory :

    === mmdiag: memory ===
    mmfsd heap size: 37949952 bytes
     
     
    Statistics for MemoryPool id 1 ("Shared Segment (EPHEMERAL)")
             128 bytes in use
      1019058540 hard limit on memory usage
          262144 bytes committed to regions
              70 allocations
              70 frees
               0 allocation failures
     
     
    Statistics for MemoryPool id 2 ("Shared Segment")
        47352608 bytes in use
      1019058540 hard limit on memory usage
        54657024 bytes committed to regions
        13114131 allocations
        13108168 frees
               0 allocation failures
     
     
    Statistics for MemoryPool id 3 ("Token Manager")
        24569728 bytes in use
       510027355 hard limit on memory usage
        27000832 bytes committed to regions
            4265 allocations
            2825 frees
               0 allocation failures

    I would like to know if some correlations exist between those 2 outputs ?

    Updated on 2013-09-12T10:50:35Z at 2013-09-12T10:50:35Z by GPFSnerd
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: Calculate GPFS memory consumption on AIX

    ‏2013-09-13T21:07:20Z  
    • GPFSnerd
    • ‏2013-09-12T10:20:55Z

    I mean how can i calculate with AIX and GPFS commands the amount of memory used on the system for GPFS ?

    I need this information because we can't figure out and bench this part of GPFS. Is it Shared Segment consumption in "mmfsadm dump fs" + Pagepool visible in SVMON + heap size in "mmdiag --memory" + The Token Memory pool 3 ? (Is the "mmfsadm dump fs" shows the actual used memory or the theorical used memory with the umalloc defined limits ?)

    If i check mmfsadm dump fs, i get :

    Filesystem dump:
      UMALLOC limits:
        bufferDescLimit      10000 desired    10000
        fileCacheLimit        1000 desired     1000
        statCacheLimit        4000 desired     4000
        diskAddrBuffLimit      800 desired      800
     
        buffDescMem          8047 k  =   10000 *  824 bytes
        fileCacheMem       878223 k  =  280330 * 3208 bytes (inode size 512 + 2696)
        statCacheMem         1281 k  =    4000 *  328 bytes
        diskAddrBufMem        113 k  =     800 *  144 bytes
        allocSegmentMem      2500 k  =   10000 *  256 bytes
        otherMem           157389 k
        mdTemp mem            256 k
     

    With mmdiag --memory :

    === mmdiag: memory ===
    mmfsd heap size: 37949952 bytes
     
     
    Statistics for MemoryPool id 1 ("Shared Segment (EPHEMERAL)")
             128 bytes in use
      1019058540 hard limit on memory usage
          262144 bytes committed to regions
              70 allocations
              70 frees
               0 allocation failures
     
     
    Statistics for MemoryPool id 2 ("Shared Segment")
        47352608 bytes in use
      1019058540 hard limit on memory usage
        54657024 bytes committed to regions
        13114131 allocations
        13108168 frees
               0 allocation failures
     
     
    Statistics for MemoryPool id 3 ("Token Manager")
        24569728 bytes in use
       510027355 hard limit on memory usage
        27000832 bytes committed to regions
            4265 allocations
            2825 frees
               0 allocation failures

    I would like to know if some correlations exist between those 2 outputs ?

    The "mmfsadm dump fs" data is the theoretical memory used for different types of objects that get cached in the Shared Segment (memory pool 2).

    The "mmfsadm dump malloc" data shows that actual usage at this point in time.  The "heap size" and the "bytes committed to regions" are the actual amount of memory that has been allocated for use.

    The pagepool is real memory that gets pinned when the daemon starts. The dump malloc data is just incrementally allocated as needed up to the hard limit set by the configuration variables (or real memory constraints) and is pageable.