Topic
  • 4 replies
  • Latest Post - ‏2012-04-03T15:21:19Z by ThinkOpenly
ThinkOpenly
ThinkOpenly
41 Posts

Pinned topic determining partition layouts

‏2012-04-02T16:40:24Z |
For inter-partition communications, there are definite advantages to having both partitions on the same POWER chip/book.  (Indeed, there are some good optimizations in the "ibmveth" driver which appear in SLES 11 SP2 and RHEL 6.2.)
 
Is there a way to determine such partition layouts?
 
"numactl --hardware" will show NUMA nodes, memory per node, and NUMA distances between nodes for a single partition, and that information can be used to determine if that partition is contained within a single NUMA node.
 
Is there a way to determine if two or more partitions are all on the same NUMA node (POWER chip/book)?  I've been referred to the AIX "lssrad" command, but I'm not sure what that does, or if there is a Linux equivalent?
Updated on 2012-04-03T15:21:19Z at 2012-04-03T15:21:19Z by ThinkOpenly
  • Bill_Buros
    Bill_Buros
    167 Posts

    Re: determining partition layouts

    ‏2012-04-02T21:11:40Z  
    Interesting question.    That's been on our "that would be good to know" list as well.     I didn't think there was any specific way to force particular cores to be assigned to a specific partition, did you already figure out how to do that?     or are you just hoping to "see" that two partitions have the same NUMA domain?
     
    We've been playing with partitions with sub-processor assignments (for example, setting the partition's entitlement to 50% of a core), and the processor specific information like caches is no longer available to the operating system.   
  • ThinkOpenly
    ThinkOpenly
    41 Posts

    Re: determining partition layouts

    ‏2012-04-02T22:42:06Z  
     With dedicated partitions (ones that use dedicated processors), the partition assignment seems to be fairly "sticky", although from what I've gathered, no "history" is maintained, so one method to "reset" assignments is to "clear the board" by shutting down everything which might be in the way (which could be everything, unless one is pretty confident about the current setup), then create and start a partition which grabs all the processors.  Shut that down, then carefully start just the ones for which placement is most critical.  (Hold off even on the VIO servers...if a partition needs it, it will stall during boot, waiting for it, but it's placement is already determined.)  A great help also is making use of affinity groups, putting whatever "sets" of partitions need to have best affinity into the same group.
  • Bill_Buros
    Bill_Buros
    167 Posts

    Re: determining partition layouts

    ‏2012-04-03T13:18:31Z  
     With dedicated partitions (ones that use dedicated processors), the partition assignment seems to be fairly "sticky", although from what I've gathered, no "history" is maintained, so one method to "reset" assignments is to "clear the board" by shutting down everything which might be in the way (which could be everything, unless one is pretty confident about the current setup), then create and start a partition which grabs all the processors.  Shut that down, then carefully start just the ones for which placement is most critical.  (Hold off even on the VIO servers...if a partition needs it, it will stall during boot, waiting for it, but it's placement is already determined.)  A great help also is making use of affinity groups, putting whatever "sets" of partitions need to have best affinity into the same group.
    In our dealings with various customers, that level of trying to force the system to "choose wisely" tends to be more work than they are ready to invest, in particular if it cannot be guaranteed in the long-run as partitions are re-started (which is rare anyway).    But I can certainly picture some select customers who would insist on the ultimate control of resources.
     
    That's an interesting observation on affinity groups.   I assume that must be an HMC or IVM feature.    Can that help keep partition resources (or multiple partitions) on a single socket?
  • ThinkOpenly
    ThinkOpenly
    41 Posts

    Re: determining partition layouts

    ‏2012-04-03T15:21:19Z  
    In our dealings with various customers, that level of trying to force the system to "choose wisely" tends to be more work than they are ready to invest, in particular if it cannot be guaranteed in the long-run as partitions are re-started (which is rare anyway).    But I can certainly picture some select customers who would insist on the ultimate control of resources.
     
    That's an interesting observation on affinity groups.   I assume that must be an HMC or IVM feature.    Can that help keep partition resources (or multiple partitions) on a single socket?
     "Can [affinity groups] help keep partition resources (or multiple partitions) on a single socket?"
     
    That is the primary purpose of affinity groups: to keep multiple partitions in physical processor/memory layouts with the least [available] latency.  I say "available", because the hypervisor cannot predict the future size or even active members of the affinity group.  I've seen it migrate shared partitions amongst physical processors to accommodate newly activated members of an affinity group.  It may migrate dedicated partitions as well.  I believe as long as the active "set" of an affinity group can fit on a single socket, the hypervisor will try really hard to put it on one (subject to other higher-priority affinity groups).