Introduction

In AIX V6.1, the scalability of WLM has been greatly extended by supporting 8192 user-defined superclasses in order to support the maximum number (8192) of active WPARs at a given time. There is also finer granularity for in the percentage-based resource-limit specification. Each value in the percentage-based limit specification (<min>%, <sMax>% and <hMax>%) now can have up to two digits after the decimal point.

According to WLM terminology, a WPAR behaves like a regular WLM class in Tier-0. The base name of the WPAR becomes its WLM class name. The wlmstat command has been enhanced to include the -@ command-line option, which reports WPAR-specific resource utilization.

WPAR resource control can be used to specify:

• The amount of memory (physical memory) and CPU resources allocated to a WPAR.
• The number of processes and threads that can be created inside a WPAR.
• The amount of virtual memory that a single process within a WPAR can consume.
• A resource set (a subset of processors) to which WPAR processes can be confined.

There are two approaches to specify CPU and memory allocation:

• Share-based. The resource each WPAR receives in this approach is a fraction of its share of resource (CPU or memory) to the total shares of all currently active WPARs.
• Percentage-based:
  • Minimum% - the minimum percentage of resources that a workload partition will be guaranteed to have if it needs the resource.
  • Soft maximum% - the maximum percentage of resources that a WPAR can have when there is contention for the resource.
  • Hard maximum% - the maximum percentage of resource sthat a WPAR can have, even if there is no contention for the resource.

The total amount of minimum% for all currently active workload partitions cannot exceed 100%.

If both shares and percentage limits are mentioned, the percentage limit takes precedence.

In almost all cases, the share-based approach to control the resources should satisfy the resource control requirements. If a minimum resource amount or a limit of the maximum amount of a resource must be set for a workload partition, then percentage-based controls need to be used.

WPAR resource control for CPU

CPU resource controls for a WPAR can be specified using the mkwpar command at the time of WPAR creation or by chwpar. The chwpar command is also effective on a running WPAR.

chwpar -R shares_CPU=<shares> <wpar name>
chwpar -R CPU=<min%>-<soft max %>, <hard max%> <wpar name>

The CPU limits specified in the command in the previous example refers to the "virtual" CPU as seen by the system. An AIX partition could be a LPAR with a dedicated CPU or a DLPAR with a fractional CPU, or it could be a non-partitioned system. In each case, the CPU utilization reported by wlmstat and CPU resource control action taken by the kernel for that WPAR would be applicable to the effective virtual CPU on that partition.

To demonstrate CPU resource control, we ran several instances of CPU-intensive programs that generate workload in a WPAR and monitored the CPU usage using vmstat and wlmstat -@ commands at the same time. We kept adding workload until vmstat reported 90%+ CPU usage.

Then we added resource controls to limit the CPU usage for the active WPAR, as follows:

             
chwpar R CPU=5%-45%,50% wpar1

As a result, in the vmstat output you can clearly see the CPU percentage usage dropping to about 40%-45% from 100%, as indicated by the "id" field. This indicates that WPAR resource control is at play.

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 0 134166 105131 0 0 0 0 0 0 22 38 191 92 0 7 0 1.00 151.5
1 0 134198 105098 0 0 0 0 0 0 20 658 187 93 1 5 0 1.00 151.5
1 0 134271 104997 0 0 0 0 0 0 14 25 182 92 0 7 0 1.00 151.5
1 0 134271 104997 0 0 0 0 0 0 13 14 184 92 0 7 0 1.00 151.5
2 0 134277 104991 0 0 0 0 0 0 18 2031 203 91 7 2 0 0.84 127.0
   <wpar cpu resource control added> 
1 0 134277 104991 0 0 0 0 0 0 18 17 184 64 0 35 0 0.46 70.1
1 0 134277 104991 0 0 0 0 0 0 9 17 181 60 0 39 0 0.43 65.6
1 0 134277 104991 0 0 0 0 0 0 9 12 176 56 0 44 0 0.40 61.0
1 0 134277 104991 0 0 0 0 0 0 18 13 179 44 0 55 0 0.32 48.7
1 0 134277 104991 0 0 0 0 0 0 8 11 179 47 0 53 0 0.34 51.3
1 0 134277 104991 0 0 0 0 0 0 17 24 187 50 0 50 0 0.36 54.3
1 0 134277 104991 0 0 0 0 0 0 22 18 196 52 0 47 0 0.38 57.
1 0 134276 104992 0 0 0 0 0 0 8 17 178 51 0 49 0 0.37 55.7
1 0 134277 104991 0 0 0 0 0 0 10 15 181 64 0 36 0 0.46 69.6
1 0 134277 104991 0 0 0 0 0 0 6 13 183 60 0 40 0 0.43 64.9
  

The following wlmstat output shows CPU usage averaging around the soft max. However, you see occasional overshoots above the hard limit, which will be explained later in the article.

# wlmstat -@ 2
CLASS CPU MEM DKIO
wpar1 42.49 2.26 0.00
TOTAL 42.49 2.26 0.00

CLASS CPU MEM DKIO
wpar1 39.83 2.26 0.42
TOTAL 39.83 2.26 0.42

CLASS CPU MEM DKIO
wpar1 36.99 2.26 0.19
TOTAL 36.99 2.26 0.19

.<data truncated>
CLASS CPU MEM DKIO
wpar1 50.19 2.26 0.33
TOTAL 50.19 2.26 0.33

.<data truncated>

CLASS CPU MEM DKIO
wpar1 36.33 2.26 0.09
TOTAL 36.33 2.26 0.09

CLASS CPU MEM DKIO
wpar1 36.06 2.26 0.04
TOTAL 36.06 2.26 0.04

CLASS CPU MEM DKIO
wpar1 37.51 2.26 0.02
TOTAL 37.51 2.26 0.02

CLASS CPU MEM DKIO
wpar1 41.01 2.26 0.00
TOTAL 41.01 2.26 0.00

The reason you see occasional overshoots above the hard limit is because of the way wlmstat works. The wlm scheduler, which by default runs every 10th of a second, gathers the CPU statistics. The CPU usage thus gathered is used by the kernel, which may schedule or de-schedule a process based on its priority, current CPU usage, and CPU limits. By default, each instantaneous value of CPU usage from each update is kept for the following 15 readings and is averaged with the other 14 readings before being displayed by wlmstat. This can potentially result in a value of slightly greater than the hard limit between WLM updates.

It is possible to improve the responsiveness of WLM by increasing the frequency at which the WLM scheduler recalculates class consumption and priority for the processor. Increasing this value causes WLM to update more frequently, thereby reducing the possibility of a process exceeding its hard limit during a short time interval. However, the trade-off is the increased overhead of WLM processing occurs.

WPAR resource control for memory

Memory resource controls for a WPAR are specified using the mkwpar command at the time of WPAR creation or by chwpar. The chwpar command is also effective on a running WPAR.

Memory resource controls are defined in shares or percentages, for example:

chwpar -R shares_memory=<shares> <wpar name>
chwpar -R memory=<min%>-<soft max %>, <hard max%> <wpar name>

The memory limits specified in the previous commands refer to the "physical" memory available in the system. For example, if the system has 1GB RAM and 512MB paging space, the following chwpar command limits the available physical (or real) memory available to all the WPAR's processes combined to a hard limit of 0.85GB.

If memory requirements continue to grow, WLM would ensure that no more new pages are allocated from the RAM; rather, there will be increased paging activity.

To verify this, we ran a memory-intensive program that does malloc() and memset() (10MB each time) in a loop inside a WPAR that had specifications of <min=8%>, <smax=10%>, <hmax=12%> for memory, which translates to 12% of 1GB= 12.3MB of physical memory. Paging space was fixed at 512MB.

To enable large heap, we compiled the program as following:

cc -bmaxdata:0x80000000 getmem.c -o getmem|outline"/>
  

With resource control ON, the program went on to allocate around 550MB. However, after the program had allocated around 100MB, we could see increased paging activity from vmstat. This happens because, as the WPAR physical memory consumption approaches its maximum limit specified by the chwpar command, resource comes into play and further memory requirements are met from the paging space.

After reaching around 550MB, the test program was killed by the kernel when it tried to memset a block. Note that in AIX, memory is not really allocated until the page is used. Though malloc may return a valid address, you don't get a page until someone tries to use it through memset ().

wlmstat reported WPAR memory utilization growing and then stabilizing just under the hard max as the WPAR consumed more and more memory.

     
# wlmstat -@ 2
 CLASS CPU MEM DKIO
  abc 0.00 1.18 0.19
TOTAL 0.00 1.18 0.19

.

CLASS CPU MEM DKIO
 abc 4.15 4.68 1.88
TOTAL 4.15 4.68 1.88

CLASS CPU MEM DKIO
 abc 4.12 7.45 0.84 
TOTAL 4.12 7.45 0.84

CLASS CPU MEM DKIO
 abc 4.11 10.22 0.38
TOTAL 4.11 10.22 0.38

CLASS CPU MEM DKIO
 abc 3.69 11.18 0.18
TOTAL 3.69 11.18 0.18

CLASS CPU MEM DKIO
 abc 2.13 11.74 2.85
TOTAL 2.13 11.74 2.85
 Memory usage stabilizing just under the hard limit 

CLASS CPU MEM DKIO
 abc 2.44 11.46 1.27
TOTAL 2.44 11.46 1.27

CLASS CPU MEM DKIO
TOTAL 5.50 5.95 2.39
 <memory usage going down as the getmem process was killed>

CLASS CPU MEM DKIO
 abc 2.07 0.07 3.12
TOTAL 2.07 0.07 3.12|outline"/>

The following code sample shows the vmstat output as the test program kept on consuming memory. After it had consumed around 100MB, there was a drastic jump in the values of pi and po fields from vmstat output, which indicates increased paging activity.

# vmstat 2 10 

System configuration: lcpu=2 mem=1024MB ent=0.66

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
0 0 151647 92760 0 0 0 0 0 0 21 53 186 3 0 96 0 0.03 4.6
0 0 154277 90130 0 0 0 0 0 0 13 42 183 2 0 98 0 0.02 2.8
0 0 159147 85260 0 0 0 0 0 0 12 36 179 3 1 96 0 0.03 4.8
0 0 164151 80256 0 0 0 0 0 0 13 49 183 3 1 96 0 0.03 4.9

<increased paging activity from here onwards>

0 1 169160 78342 0 3 1335 1551 12999 1 183 36 188 3 6 76 15 0.08 12.8
0 0 171664 77827 0 27 1024 1024 1513 0 110 31 247 2 3 88 7 0.04 6.7
0 0 176674 78175 0 0 2677 2679 3478 0 194 42 192 3 7 84 6 0.08 12.4
0 1 181684 77261 0 0 2048 2048 2778 0 151 34 183 3 5 87 5 0.07 10.7
0 0 185493 77288 0 16 1936 2734 4451 0 156 44 221 2 6 85 7 0.07 10.5
0 0 189199 77225 0 0 1821 1024 1024 0 129 32 184 2 4 89 4 0.06 8.5
3 1 194208 77471 0 0 2624 2627 13102 1 190 37 178 3 8 83 6 0.10 15.4

An increased paging activity indicates performance degradation. If the paging space gets exhausted, the process requesting more memory is killed. It is therefore recommended that you cap WPAR's memory limits using shares rather than percentage limits. In that way, a WPAR would be able to use all of system's physical memory if there was no contention for memory.

However, it may be a good idea to specify a hard limit (at a high, though, of around 90%). If a faulty application keeps requesting memory without a hard max, it exhausts all the virtual memory and thereby possibly forces a system reboot.

It is also practical to specify a minimum limit for WPARs hosted in the same global environment so that applications running in them are not memory starved, even if there are other WPARs hosting memory-intensive applications.

Unlike CPU statistics, the memory stats do not cross the assigned hard limits. The reason for this is that memory is a "real" (or let's say non-virtual) entity, the statistics about memory consumed is maintained by the kernel and is always current. The WLM scheduler just reads the kernel data and reports it. On the other hand, in the case of CPU, the input from the WLM scheduler about CPU usage is read by the kernel to make scheduling decisions. So there is a scope of timing delays between the wlmstat-reported data and actual CPU utilization.

Resource control using a resource set (rset)

A resource set (rset) is used to define a subset of processors in a system. If a resource set is specified for a workload partition, it can only use the processors within the specified resource set.

This feature can be used to host compute-intensive or HPC-type applications inside a WPAR environment in order to confine certain processes to a particular set of processors. This ensures that certain applications don't end up exhausting all of the computing resources in a system.

A new rset is created by the mkrset c command. lsrset a displays all the rsets existing on a system. A WPAR can be associated with a rset with the mkwpar command when the WPAR is created or through chwpar.

# lsrset -a
sys/sys0
sys/node.01.00000
sys/mem.00000
sys/cpu.00000
sys/cpu.00001
#

The above example shows two default CPU rsets already existing on the system. The two rsets correspond to the two CPU cores of a SMT-enabled system. For the sake of simplicity, we call them logical CPU 0 and logical CPU 1.

The following command binds all the processes of WPAR "abc" with the rset sys/cpu.00001.

 chwpar rset=sys/cpu.00001 abc
    

As a result of this command, all processes from the WPAR are restricted to logical CPU 1. Logical CPU 1 may also run processes from the global. It may also run processes from other WPARs if other WPARs choose to bind their rsets with logical CPU 1.

The percentage of CPU utilization of the rset/logical cpu is reported by the mpstat command. You can see this in the following example, which reports logical CPU usage at two-second intervals.

When the system is mostly idle:

# mpstat 2
      
System configuration: lcpu=2 ent=0.7 mode=Uncapped
cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs
0 8 0 0 261 180 90 0 0 100 44 9 63 0 27 0.01 0.9 325
1 0 0 0 93 0 0 0 0 - 0 0 16 0 84 0.00 0.4 298
U - - - - - - - - - - - - 0 99 0.65 98.7 -
ALL 8 0 0 354 180 90 0 0 100 44 0 1 0 99 0.01 1.3 623
--------------------------------------------------------------------------------
0 1 0 0 261 177 88 0 0 100 21 5 61 0 34 0.00 0.7 321
1 0 0 0 89 0 0 0 0 - 0 0 18 0 82 0.00 0.3 293U - - - - - - - - - - - - 0 99 0.65 99.0 -
ALL 1 0 0 350 177 88 0 0 100 21 0 0 0 99 0.01 1.0 614
    

The columns to observe are "us," "sy," and "id," which denotes CPU utilization time in user space, system space, and system idle time in percentage. It clearly suggests that the system is 99% idle.

Now start a WPAR called "abc" with no resource control and add some workload to it by running the "yes" command inside WPAR.

         
/usr/bin/yes > dev/null &

On the mpstat command window, you can observe the following:

    cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs
0 8 0 0 268 169 75 0 0 100 3856 99 1 0 0 0.92 91.9 100
1 0 0 0 90 35015 35015 0 0 - 0 0 0 0 100 0.08 8.1 186
ALL 8 0 0 358 35184 35090 0 0 100 3856 91 1 0 8 1.00 151.7 286------
0 1 0 0 304 168 76 0 0 100 3838 99 1 0 0 0.92 91.9 100
1 0 0 0 98 34939 34939 0 0 - 0 0 0 0 100 0.08 8.1 192
ALL 1 0 0 402 35107 35015 0 0 100 3838 91 1 0 8 1.00 151.5 292
    

rset 0 is 0% idle (using 100% CPU).

Now, add resource control to the WPAR to bind it to rset 1.

   
chwpar -R rset=sys/cpu.00001 abc

And on the mpstat command window, you can observe the following:

cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs
0 8 0 0 168 180 90 0 0 100 44 0 2 0 98 0.05 5.0 242
1 0 0 0 116 2 2 0 0 100 3920 99 1 0 0 0.95 95.0 100
ALL 8 0 0 284 182 92 0 0 100 3964 95 1 0 5 1.00 151.5 342
--------------------------------------------------------------------------------
0 1 0 0 168 174 88 0 0 100 14 0 1 0 99 0.05 4.9 241
1 0 0 0 121 1 1 0 0 100 3920 99 1 0 0 0.95 95.1 100
ALL 1 0 0 289 175 89 0 0 100 3934 95 1 0 5 1.00 151.5 341

rset 0 is now idle and rset 1 is now getting all of the workload. That is because WPAR resource control has tied up the WPAR's processes to logical CPU 1. Also note the "pc" and "%ec" fields in the above mpstat output (which stands for physical CPU and effective CPU percentage utilization). The workload in logical CPU 0 accounts for only 5% of physical CPU, while logical CPU 1 accounts for the rest.

rsets can be shared among WPARs. A WPAR can also use a rset already defined for another WPAR. (although mkwpar/chwpar throws a warning).

In the continuing example, while rset 1 was fully loaded, we started another WPAR (called def) without resource control and added CPU workload to it.

In the mpstat command window, you can observe the following:

cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs
0 52 0 1 276 204 82 0 0 100 3564 99 1 0 0 0.50 50.0 100
1 0 0 1 127 0 0 0 0 100 3201 100 0 0 0 0.50 50.0 100
ALL 52 0 2 403 204 82 0 0 100 6765 99 1 0 0 1.00 151.6 200
--------------------------------------------------------------------------------
0 2 0 0 276 173 79 0 0 100 3209 99 1 0 0 0.50 50.1 100
1 0 0 0 133 3 2 0 0 100 3208 100 0 0 0 0.50 49.9 100
ALL 2 0 0 409 176 81 0 0 100 6417 100 0 0 0 1.00 151.5 200

From this example, both of the logical CPUs could be seen to completely loaded. Note the "pc" and "%ec" fields in the mpstat output. Each logical CPU is using 50% of the total physical CPU allocated to the partition.

Now bring WPAR "def" under resource control, under the same rset as the other WPAR.

# chwpar -R rset=sys/cpu.00001 def

**********************************************************************
Warning
chwpar: 
0960-105 resources.rset must be unique across all 
workload partitions.rset = sys/cpu.00001 
also found in the following files:/etc/wpars/abc.cf
**********************************************************************
  

mpstat now reports (below) that logical CPU 0 is back to an idle state, while CPU 1 takes over the workload from CPU 0 for WPAR "def"'s processes.

 
cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs
0 8 0 0 218 336590 336491 0 0 100 45 0 0 0 100 0.38 38.1 99
1 0 0 0 122 99 99 0 0 100 3142 100 0 0 0 0.63 63.3 99
ALL 8 0 0 340 336689 336590 0 0 100 3187 63 0 0 37 1.01 153.6 198
--------------------------------------------------------------------------------
0 1 0 0 182 339612 339514 0 0 100 13 0 0 0 100 0.37 36.7 100

1 0 0 0 125 99 99 0 0 100 3166 100 0 0 0 0.63 63.3 100
ALL 1 0 0 307 339711 339613 0 0 100 3179 63 0 0 37 1.00 151.5 200

It may also be seen that physical CPU usage is now tilted in favor of logical CPU 1 (63.3%), as indicated by the fields "pc" and "%ec."

In the previous rset examples, you observed that the CPU usage was maxed out. It is also possible to combine CPU limits (shares or percentage based) to a rset to limit the CPU usage in a rset. When a CPU limit is applied to a rset, the limit would apply to the total total available CPU resources in that rset.

Back to top

Exclusive rsets

It is also possible to assign exclusive rsets to WPARs. When a WPAR is attached to an exclusive rset, processes from the WPAR alone are allowed to run on the CPUs belonging to that rset. This is useful in cases when a user wants to assign exclusive computing resources for CPU-intensive applications.

To create an exclusive rset, run:

         
mkrset -c 1 sysxrset/<rset name>

To attach the WPAR to the exclusive rset, run:

  
chwpar -R rset=sysxrset/<rset name> <wpar name>

Currently, an exclusive rset can be associated to more than one WPAR, although it might change in the future.

Please note that CPU limits are not applicable to exclusive rsets. If CPU limits are mentioned, they are ignored.

Back to top

Conclusion

Systems administrators can effectively control the usage of resources across WPARs by planning beforehand the preference and the resource share that should be given to each of the WPARs hosted in a system when creating a WPAR using the mkwpar command. Also, they can fine tune the performance on-the-fly when the workload changes by using the chwpar command.

Resources

Learn

• The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX? Visit the New to AIX and UNIX page to learn more.
  
  
• Browse the technology bookstore for books on this and other technical topics.
  
  
• The Introduction to Workload Partition Management in IBM AIX Version 6.1 Redbook introduces the new feature introduced in AIX 6.1.
  
  
• The AIX 5L Workload Manager (WLM) Redbook covers the WLM feature and tools and is intended to be a reference to system administrators.
  
  

Discuss

• Check out developerWorks blogs and get involved in the developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX Forum
  • AIX Forum for developers
  • Cluster Systems Management
  • IBM Support Assistant Forum
  • Performance Tools Forum
  • Virtualization Forum
  • More AIX and UNIX Forums
  
  

About the authors

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Exclusive rsets
• Conclusion
• Resources
• About the authors
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers