Topic
  • 14 replies
  • Latest Post - ‏2013-05-10T15:44:21Z by manwerjalil
k0da
k0da
7 Posts

Pinned topic Power5 and amount of memory assigned

‏2011-06-16T13:23:29Z |
Hi,
I'm running SLES11SP1 and have confused with amount of memory that kernel detects is lower, that partition have assigned.
HMC reports, I have 1GB memory assigned for LPARs
~> lshwres -r mem -m shark --level lpar --filter "lpar_ids=7"
lpar_name=07-sunrise,lpar_id=7,curr_min_mem=768,curr_mem=1024,curr_max_mem=1024,pend_min_mem=768,pend_mem=1024,pend_max_mem=1024,run_min_mem=768,run_mem=1024

But OS shows only 875M
  1. cat /proc/meminfo | grep MemTotal
MemTotal: 896832 kB
  1. free -k
total used free shared buffers cached
Mem: 896832 808384 88448 0 268224 373440
-/+ buffers/cache: 166720 730112
Swap: 1534016 107456 1426560
  1. uname -r
2.6.32.36-0.5-ppc64

Other partitions have 1GB memory as well, but OS there reports different numbers.
Another example:
~> lshwres -r mem -m shark --level lpar --filter "lpar_ids=10"
lpar_name=09-kiwipower,lpar_id=10,curr_min_mem=768,curr_mem=1024,curr_max_mem=1024,pend_min_mem=768,pend_mem=1024,pend_max_mem=1024,run_min_mem=768,run_mem=1024
  1. cat /proc/meminfo | grep MemTotal
MemTotal: 1018496 kB
  1. free -k
total used free shared buffers cached
Mem: 1018496 240000 778496 0 40000 88256
-/+ buffers/cache: 111744 906752
Swap: 7872 0 7872

Is that behavior normal?
Updated on 2011-06-23T09:57:24Z at 2011-06-23T09:57:24Z by k0da
  • jscheel
    jscheel
    31 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-17T15:51:30Z  
    Yes, what you are observing is that the Hypervisor is using memory on behalf of the partition. That means what one allocates for memory on the HMC never reflects what's actually in the partition as shown by the OS. If you are concerned about what's technically going on, I suspect we'll find more details in a Redbook. I'm just unsure in which one.

    However, I can with confidence tell you that you are not imagining things, nor have you stumbled on a defect. It's working as designed.

    Hope that helps. Let me know if you want more information and I'll track that down for you.
    -Jeff
  • k0da
    k0da
    7 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-17T16:11:12Z  
    • jscheel
    • ‏2011-06-17T15:51:30Z
    Yes, what you are observing is that the Hypervisor is using memory on behalf of the partition. That means what one allocates for memory on the HMC never reflects what's actually in the partition as shown by the OS. If you are concerned about what's technically going on, I suspect we'll find more details in a Redbook. I'm just unsure in which one.

    However, I can with confidence tell you that you are not imagining things, nor have you stumbled on a defect. It's working as designed.

    Hope that helps. Let me know if you want more information and I'll track that down for you.
    -Jeff
    I will appreciate, if you provide me with Redbook name.

    I though, that memory should be assigned as is or at least with no such big drawbacks
  • jscheel
    jscheel
    31 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-17T18:15:22Z  
    • k0da
    • ‏2011-06-17T16:11:12Z
    I will appreciate, if you provide me with Redbook name.

    I though, that memory should be assigned as is or at least with no such big drawbacks
    Would you mind doing a "cat /proc/ppc64/lparcfg" and telling me what the DesMem field says? I suspect it matches the value for which you are looking, but please confirm.
  • jscheel
    jscheel
    31 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-17T18:17:57Z  
    • jscheel
    • ‏2011-06-17T18:15:22Z
    Would you mind doing a "cat /proc/ppc64/lparcfg" and telling me what the DesMem field says? I suspect it matches the value for which you are looking, but please confirm.
    If you're looking for information on all of the lparcfg fields, it can be found here: http://www.ibm.com/developerworks/wikis/display/LinuxP/lparcfg+documentation
  • k0da
    k0da
    7 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-17T18:19:51Z  
    • jscheel
    • ‏2011-06-17T18:15:22Z
    Would you mind doing a "cat /proc/ppc64/lparcfg" and telling me what the DesMem field says? I suspect it matches the value for which you are looking, but please confirm.
    1. cat /proc/ppc64/lparcfg | grep DesMem
    DesMem=1024
  • jscheel
    jscheel
    31 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-20T13:59:22Z  
    • k0da
    • ‏2011-06-17T18:19:51Z
    1. cat /proc/ppc64/lparcfg | grep DesMem
    DesMem=1024
    Ok, as expected. This matches your "cur_mem" field from the HMC. So, it appears the partition has seen all the memory but Linux is not totalling it as available.

    We'll dig deeper and see what we can find out. I still suspect this is working as designed but may not be documented. Once we have a definitive, technical answer, we can correct the documentation part. Thanks for your patience.
  • AntonBlanchard
    AntonBlanchard
    4 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-21T22:14:49Z  
    Run dmesg and look for the line with physicalMemorySize, eg:

    physicalMemorySize = 0xf30000000

    This should be the amount of memory the hypervisor has given us. If they are the same, then Linux is responsible for the difference.

    One of the big consumers of memory at boot for Linux is kdump. Check /proc/cmdline on both boxes to make sure they have the same kdump settings.
  • k0da
    k0da
    7 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-22T13:56:33Z  
    Run dmesg and look for the line with physicalMemorySize, eg:

    physicalMemorySize = 0xf30000000

    This should be the amount of memory the hypervisor has given us. If they are the same, then Linux is responsible for the difference.

    One of the big consumers of memory at boot for Linux is kdump. Check /proc/cmdline on both boxes to make sure they have the same kdump settings.
    I have physicalMemorySize = 0x40000000, so if I had converted correctly it is 1Gb.

    So Kernel bug?
  • k0da
    k0da
    7 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-22T13:57:53Z  
    • k0da
    • ‏2011-06-22T13:56:33Z
    I have physicalMemorySize = 0x40000000, so if I had converted correctly it is 1Gb.

    So Kernel bug?
    Ah, forgot to mention. I have no kdump configured
  • MikeWolf
    MikeWolf
    2 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-22T14:44:28Z  
    I have configured a partition with 1 GB of memory and installed SLES11SP1 on it.
    MemTotal: 896576 kB
    DesMem=1024
    root=/dev/sda4 quiet sysrq=1 insmod=sym53c8xx insmod=ipr crashkernel=512M-:128M

    which is nearly identical to what you are reporting. If I take the "crashkernel=" off
    of the cmdline I then see:
    MemTotal: 1016832 kB
    DesMem=1024
    root=/dev/sda4 quiet sysrq=1 insmod=sym53c8xx insmod=ipr

    which is similiar to the other system you compared to. Please post the kernel commandline
    'cat /proc/cmdline' and double check about crashkernel on it. SLES configures that by default
  • k0da
    k0da
    7 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-22T15:04:24Z  
    • MikeWolf
    • ‏2011-06-22T14:44:28Z
    I have configured a partition with 1 GB of memory and installed SLES11SP1 on it.
    MemTotal: 896576 kB
    DesMem=1024
    root=/dev/sda4 quiet sysrq=1 insmod=sym53c8xx insmod=ipr crashkernel=512M-:128M

    which is nearly identical to what you are reporting. If I take the "crashkernel=" off
    of the cmdline I then see:
    MemTotal: 1016832 kB
    DesMem=1024
    root=/dev/sda4 quiet sysrq=1 insmod=sym53c8xx insmod=ipr

    which is similiar to the other system you compared to. Please post the kernel commandline
    'cat /proc/cmdline' and double check about crashkernel on it. SLES configures that by default
    Yes, got 994MB by disabling kdump on SLES LPAR.

    Ok, but where 30M goes to?

    I'll appreciate, if somebody can provide RedBook name where I can read about it.
  • MikeWolf
    MikeWolf
    2 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-22T15:49:38Z  
    Some of it is used by the Run Time Abstraction Service (rtas)
    some is taken up by OF-lite. To see the size rtas is taking
    up.

    od -x /proc/device-tree/rtas/rtas-size
  • k0da
    k0da
    7 Posts

    Re: Power5 and amount of memory assigned

    ‏2011-06-23T09:57:24Z  
    • MikeWolf
    • ‏2011-06-22T15:49:38Z
    Some of it is used by the Run Time Abstraction Service (rtas)
    some is taken up by OF-lite. To see the size rtas is taking
    up.

    od -x /proc/device-tree/rtas/rtas-size
    Thanks, now it is clear for me.
  • manwerjalil
    manwerjalil
    2 Posts

    Re: Power5 and amount of memory assigned

    ‏2013-05-10T15:44:21Z  

    Guys I am in same trouble. I have one IBM p701 with 16GB memory running PPC64 linux. This is physical machine and it shows only 2GB memory out of 16GB. I have reinstalled this server again to resolve this issue but still same problem.

    Please help me.