Chris's AIX Blog

Starting with AIX 6.1 TL08 and AIX 7.1 TL02 there’s a new AIX CPU tuning feature called “Scaled Throughput” mode. This is supported on POWER7 and POWER7+ processors only (do not try this on POWER6!). This new mode has the ability to dispatch workload to more SMT threads per VP, avoiding the need to unfold additional VPs. I’ve heard it described as being more “POWER6 like”. I’m not suggesting that you use this feature. This post simply discusses what this new mode can do.

 

By default AIX (on POWER7) operates in “Raw Throughput” mode. This mode provides the best performance per thread per core. It offers the best response times. It utilises more cores (VPs) to process a systems workload. By comparison, “Scaled Throughput” provides a greater level of per core throughput (processing) by dispatching more SMT threads on a core. This has the effect of utilising fewer VPs/cores. In this mode, more (or all) SMT threads per core will be utilised before dispatching workload to other VPs/core in the system.

 

The schedo tuning command can be used to enable the new mode using a new parameter called vpm_throughput_mode. e.g.

 

# schedo –p –o vpm_throughput_mode=X

 

This tunable can be set to one of the following values:

 

0 = Legacy Raw mode (default).

1= Scaled or “Enhanced Raw” mode with a higher threshold than legacy.

2 = Scaled mode, use primary and secondary SMT threads.

4 = Scaled mode, use all four SMT threads.

 

At this stage, this tunable is not restricted, but if you plan on experimenting with it please be careful; make sure you understand how this tuning may impact your system; always test new tuning in a non-production environment first!

 

I performed a couple of quick tests today, just to see what impact tuning the parameter would have on an AIX 7.1 TL2 system.

 

aixlpar1 : / # oslevel -s

7100-02-01-1245

 

aixlpar1 : / # lsconf | grep Mode

System Model: IBM,9119-FHB

Processor Implementation Mode: POWER 7

 

I started some CPU intensive workload.

 

Name           PID  CPU%  PgSp Owner            CliV2         0  WPAR Total    0

ncpu        4718826 15.7  108K db               SerV3         0  Press: "h"-help

ncpu        7143668 15.7  108K db               CliV3         0         "q"-quit

ncpu        8126488 15.7  108K db

ncpu        5832920 15.6  108K db

 

The vpm_throughput_mode parameter was left at the default value (0).

 

# schedo -a | grep vpm_throughput_mode

          vpm_throughput_mode = 0

 

 

As expected, the workload was evenly dispatched across each of the primary SMT threads of the 4 VPs assigned to the partition i.e. logical CPU 0, 4, 8 and 12. None of the secondary or tertiary SMT threads were active. This is the default mode and will provide the greatest raw throughput (performance) per VP (as there’s no overhead associated with enabling secondary or tertiary SMT threads).

 

Topas Monitor for host:aixlpar1              EVENTS/QUEUES    FILE/TTY

Fri Dec 21 11:06:34 2012   Interval:2           Cswitch     184  Readch      864

                                                Syscall     299  Writech     651

CPU     User% Kern% Wait% Idle%   Physc         Reads       111  Rawin         0

0        99.7   0.3   0.0   0.0    0.63         Writes        1  Ttyout      233

1         0.2   0.5   0.0  99.3    0.12         Forks         0  Igets         0

2         0.0   0.0   0.0 100.0    0.12         Execs         0  Namei         9

3         0.0   0.0   0.0 100.0    0.12         Runqueue   4.00  Dirblk        0

4       100.0   0.0   0.0   0.0    0.63         Waitqueue   0.0

5         0.0   0.0   0.0 100.0    0.12                          MEMORY

6         0.0   0.0   0.0 100.0    0.12         PAGING           Real,MB    4096

7         0.0   0.0   0.0 100.0    0.12         Faults        0  % Comp     22

8       100.0   0.0   0.0   0.0    0.63         Steals        0  % Noncomp   2

9         0.0   0.0   0.0 100.0    0.12         PgspIn        0  % Client    2

10        0.0   0.0   0.0 100.0    0.12         PgspOut       0

11        0.0   0.0   0.0 100.0    0.12         PageIn        0  PAGING SPACE

12      100.0   0.0   0.0   0.0    0.63         PageOut       0  Size,MB    2048

13        0.0   0.0   0.0 100.0    0.12         Sios          0  % Used      0

14        0.0   0.0   0.0 100.0    0.12                          % Free    100

15        0.0   0.0   0.0 100.0    0.12         NFS (calls/sec)

 

Next, we enabled scaled throughput mode (2). The workload slowly migrated to logical CPUs 4, 5, 8 and 9.  So now only two primary SMT threads were active (lcpu’s 4 and 8) and two secondary threads were active (lcpu’s 5 and 9). All the processing was being performed by fewer VPs (almost like POWER6).

 

# schedo -p -o vpm_throughput_mode=2

 

Topas Monitor for host:aixlpar1              EVENTS/QUEUES    FILE/TTY

Fri Dec 21 11:07:36 2012   Interval:2           Cswitch     179  Readch      935

                                                Syscall     301  Writech     794

CPU     User% Kern% Wait% Idle%   Physc         Reads       112  Rawin         0

0        14.1  60.3   0.0  25.6    0.00         Writes        2  Ttyout      304

1         5.7  32.8   0.0  61.5    0.00         Forks         0  Igets         0

2         0.0   1.6   0.0  98.4    0.00         Execs         0  Namei        10

3         0.0   3.0   0.0  97.0    0.00         Runqueue   4.00  Dirblk        0

4       100.0   0.0   0.0   0.0    0.47         Waitqueue   0.0

5       100.0   0.0   0.0   0.0    0.47                          MEMORY

6         0.0   0.0   0.0 100.0    0.03         PAGING           Real,MB    4096

7         0.0   0.0   0.0 100.0    0.03         Faults        0  % Comp     22

8       100.0   0.0   0.0   0.0    0.47         Steals        0  % Noncomp   2

9       100.0   0.0   0.0   0.0    0.47         PgspIn        0  % Client    2

10        0.0   0.0   0.0 100.0    0.03         PgspOut       0

11        0.0   0.0   0.0 100.0    0.03         PageIn        0  PAGING SPACE

12        0.0  50.6   0.0  49.4    0.00         PageOut       0  Size,MB    2048

15        0.0  13.3   0.0  86.7    0.00         Sios          0  % Used      0

14        0.0   0.8   0.0  99.2    0.00                          % Free    100

15        0.0   0.8   0.0  99.2    0.00         NFS (calls/sec)

 

And finally, I tried scaled mode with all four SMT threads (4). All of the workload was migrated to a single VP but all 4 SMT threads were being utilised (primary SMT thread lcpu 0, secondary/tertiary SMT threads 1, 2 &3). This mode offers lower overall core consumption but has the (possibly negative) side effect of enabling more SMT threads on a single VP/core, which may not perform as well as the same workload evenly dispatched to 4 individual VPs/cores (on primary SMT threads).

 

# schedo -p -o vpm_throughput_mode=4

 

Topas Monitor for host:aixlpar1              EVENTS/QUEUES    FILE/TTY

Fri Dec 21 11:08:30 2012   Interval:2           Cswitch     199  Readch      385

                                                Syscall     149  Writech     769

CPU     User% Kern% Wait% Idle%   Physc         Reads         2  Rawin         0

0        99.7   0.3   0.0   0.0    0.25         Writes        1  Ttyout      375

1        99.9   0.1   0.0   0.0    0.25         Forks         0  Igets         0

2       100.0   0.0   0.0   0.0    0.25         Execs         0  Namei         6

3       100.0   0.0   0.0   0.0    0.25         Runqueue   4.00  Dirblk        0

4         0.0  48.8   0.0  51.2    0.00         Waitqueue   0.0

5         0.0   5.5   0.0  94.5    0.00                          MEMORY

6         0.0   3.6   0.0  96.4    0.00         PAGING           Real,MB    4096

7         0.0   3.3   0.0  96.7    0.00         Faults        0  % Comp     22

8         0.0  77.1   0.0  22.9    0.00         Steals        0  % Noncomp   2

9         0.0  52.9   0.0  47.1    0.00         PgspIn        0  % Client    2

10        0.0  44.8   0.0  55.2    0.00         PgspOut       0

11        0.0  38.5   0.0  61.5    0.00         PageIn        0  PAGING SPACE

12        0.0  51.7   0.0  48.3    0.00         PageOut       0  Size,MB    2048

13        0.0   8.9   0.0  91.1    0.00         Sios          0  % Used      0

14        0.0   7.0   0.0  93.0    0.00                          % Free    100

15        0.0   7.4   0.0  92.6    0.00         NFS (calls/sec)

 

For more information on “Scaled Throughput” mode, take a look at the following presentation:

 

http://t.co/jVtJ40Ty

 

 

In a previous post I discussed how you can identify some of the different types of a PowerVM Capacity on Demand (CoD) activation keys from IBM.

 

Recently I had to Activate Memory Expansion (AME) on a couple of POWER7 systems. I discovered that all of the keys contained a similar string. It appears that if a CoD key contains the string CA1F0000000800 then it is safe to assume it will activate AME for a particular system. e.g.  

 

9741EF3AE6969F17CA1F0000000800419D 

937A1240F00F5B05CA1F0000000800413D

 

And while I’m talking about AME, I thought I’d share this tip as well.

 

I was performing a demo of AME for my team and wanted to change the AME expansion factor using DLPAR during the demo. I did not want to use the HMC GUI but rather the HMC command line (as it’s faster).

 

To change the expansion factor for an LPAR (that’s enabled for AME), you can use the chhwres command from the HMC CLI.

 

During the demo I highlighted the current (running) expansion factor for the LPAR (using the lshwres command).

 

hscroot@hmc1:~> lshwres -m 795-1 -r mem --level lpar --filter "lpar_ids=189"

lpar_name=aixlpar1,lpar_id=89,curr_min_mem=1024,curr_mem=4096,curr_max_mem=32768,pend_min_mem=1024,pend_mem=4096,pend_max_mem=32768,run_min_mem=1024,run_mem=4096,curr_min_num_huge_pages=0,curr_num_huge_pages=0,curr_max_num_huge_pages=0,pend_min_num_huge_pages=0,pend_num_huge_pages=0,pend_max_num_huge_pages=0,run_num_huge_pages=0,mem_mode=ded,curr_mem_expansion=2.0,pend_mem_expansion=2.0,curr_hpt_ratio=1:64,curr_bsr_arrays=0

 

Then I increased the expansion factor from 2.0 to 3.0 (using DLPAR).

 

hscroot@hmc1:~> chhwres -r mem -m 795-1 -o s -p aixlpar1 -a "mem_expansion=3"

 

Using the lshwres command I then confirmed that the expansion factor changed (from the HMC command line).

 

hscroot@hmc1:~> lshwres -m 795-1 -r mem --level lpar --filter "lpar_ids=189"

lpar_name=aixlpar1,lpar_id=89,curr_min_mem=1024,curr_mem=4096,curr_max_mem=32768,pend_min_mem=1024,pend_mem=4096,pend_max_mem=32768,run_min_mem=1024,run_mem=4096,curr_min_num_huge_pages=0,curr_num_huge_pages=0,curr_max_num_huge_pages=0,pend_min_num_huge_pages=0,pend_num_huge_pages=0,pend_max_num_huge_pages=0,run_num_huge_pages=0,mem_mode=ded,curr_mem_expansion=3.0,pend_mem_expansion=3.0,curr_hpt_ratio=1:64,curr_bsr_arrays=0

 

I also ran the lparstat command on the LPAR (before and after the DLPAR operation) to demonstrate the changed expansion factor value and the increase in expanded memory size.

 

aixlpar1 : / # lparstat -i | grep Exp

Memory Mode                                : Dedicated-Expanded

Target Memory Expansion Factor             : 2.00

Target Memory Expansion Size               : 8192 MB

 

aixlpar1 : /# lparstat -i | grep Exp

Memory Mode                                : Dedicated-Expanded

Target Memory Expansion Factor             : 3.00

Target Memory Expansion Size               : 12288 MB

 

Here are some questions I received recently regarding VLAN tagging on the VIO server. My answers are shown in green.

 

“Hi Chris,

 

Q: I’m trying to understand when, where and why there would be the need to use ‘mkvdev –vlan (etc.) on the VIOS, and I’m wondering whether you would be able to clarify this for me, please.

 

Is it necessary to add the VLAN tag devices to the SEA, or is it suffice to just have them defined within the Virtual Ethernet itself which is part of the SEA?”

 

A: It is suffice to simply define the VLAN ids assigned to the Virtual Ethernet adapters associated with the SEA.

 

“Q: For completeness, on the rare occasions I have done this, I have added the VLAN’s to the Virtual Ethernet and also as VLAN devices on the VIOS (mkvdev –vlan etc.)”

 

A:  mkvdev –vlan is not necessary, unless the VIOS needs to communicate with hosts on different VLANs i.e. you need an IP address on the VIOS for each VLAN. This does not mean the SEA will bridge this VLAN traffic for VIOCs.

 

“Q: The reason I started thinking of this is, is because one of our customers wants to add new VLAN’s to their SEA, but they’re not running Power7 hardware. Therefore, the online method would be to add a new Virtual Adapter which contains the new VLAN ID’s to the VIOS using DLPAR, then use chdev –dev (etc.) on the SEA to include the new Virtual Ethernet.”

 

A: Agreed. The “IBM PowerVM Virtualization Managing and Monitoring” Redbook states: “If your system doesn’t support dynamic VLAN modifications and you are modifying the VLAN list of a virtual Ethernet adapter that is configured in a SEA with ha_mode enabled, the HMC will not allow you to reconfigure the list of VLANs on that interface. You will need to add an additional virtual Ethernet adapter and modify the virt_adapters list of the SEA, or modify the profile of both Virtual I/O Servers and re-activate both Virtual I/O Servers at the same time.”

 

“Q: From the phone call I had, it would appear that the VLAN tags are included on the Virtual Ethernet device, but have not been added to the SEA by running mkvdev –vlan (etc. ) on the VIOS’s.  This leads me to assume that the ‘mkvdev –vlan’ is only required if there is a requirement to access the VIOS itself from a particular VLAN. Am I right, or is there something I’m not understanding? I’m unable to find documentation that explains the answer. Do you happen to know?”

 

A: That is also my understanding (based on my experience). On page 483 of the “IBM PowerVM Virtualization Introduction and Configuration” Redbook , it states: “The addition of VLAN interfaces to the SEA adapter is only necessary if the VIO Server itself needs to communicate on these VLANs”.

 

“Q: Hi Chris,

We are trying to associate a new entX Virtual Ethernet Trunk Device to an existing SEA. The new device must be configured for VLAN tagging. The existing virtual Ethernet adapter that (is already associated with the SEA) is not configured for VLAN tagging. This device will remain associated to the SEA and continue to pass untagged packets to the already configured network.

 

Ultimately  the configuration we want would be two entX devices associated with the existing SEA. One entX device is configured for notagged packets and the other entX device is configured for tagging.

 

How do we configured this?”

 

A: What you are trying to do should work:

 

Existing setup:

 

VIOC1 boot1 PVID 20 ---> VIOS1 ent3 PVID 20 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X

VIOC1 boot2 PVID 21 ---> VIOS2 ent3 PVID 21 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X

 

Proposed setup:

 

VIOC1 boot1 PVID 20 ---> VIOS1 ent3 PVID 20 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X (NO CHANGE)

VIOC1 boot2 PVID 21 ---> VIOS2 ent3 PVID 21 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X (NO CHANGE)

 

VIOC2 boot1 PVID 22 ---> VIOS1 ent6 PVID 92*, VID 22, 802.1Q   ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X, VID=22 (NEW)

VIOC2 boot2 PVID 23 ---> VIOS2 ent6 PVID 93*,  VID 23, 802.1Q  ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X, VID=23  (NEW)

 

*The PVID can be any unused/throw away number. The new ent6 adapter would be configured similar to this:

 

 

Then you would run this command to associate the new ent6 with the existing SEA, ent5:

 

$ chdev -dev ent5 -attr virt_adapters=ent3,ent6

 

Have a look at the following sections of this Redbook:

 

3.2.3 Virtual I/O Server: Dynamically add or modify an existing adapter such that the required VLAN is listed in the 802.1q Additional VLAN fields (on POWER7 only).

 

3.2.2 Hardware Management Console: Dynamic VLAN modification in the GUI (on POWER7 only).

 

http://www.redbooks.ibm.com/redpieces/abstracts/sg247590.html?Open

 

This document (by Anthony English) is also a very useful document relating to VLANs, SEAs and VIOS:

 

http://www.ibm.com/developerworks/aix/library/au-managevlans/index.html?ca=drs-

 

 

Reply: “hmm ok I see what you are saying, I will give it a go and tell you how it turns out...thanks. ok finally got around to testing using a VIOS at DR site. Created new virtual adapter PVID 55 and VID 888 (ent9) then added it to the existing SEA as shown below:

 

 

accounting    enabled   Enable per-client accounting of network statistics                 True
ctl_chan      ent6      Control Channel adapter for SEA failover                           True
gvrp          no        Enable GARP VLAN Registration Protocol (GVRP)                      True
ha_mode       auto      High Availability Mode                                             True
jumbo_frames  no        Enable Gigabit Ethernet Jumbo Frames                               True
large_receive no        Enable receive TCP segment aggregation                             True
largesend     0         Enable Hardware Transmit TCP Resegmentation                        True
lldpsvc       no        Enable IEEE 802.1qbg services                                      True
netaddr       0         Address to ping                                                    True
pvid          40        PVID to use for the SEA device                                     True
pvid_adapter  ent4      Default virtual adapter to use for non-VLAN-tagged packets         True
qos_mode      disabled  N/A                                                                True
real_adapter  ent2      Physical adapter associated with the SEA                           True
thread        1         Thread mode enabled (1) or disabled (0)                            True
virt_adapters ent4,ent9 List of virtual adapters associated with the SEA (comma separated) True

 

VLAN Ids :
Invalid VLAN ID Packets: 0
Port VLAN ID:    40
VLAN Tag IDs:  None
Invalid VLAN ID Packets: 0
Port VLAN ID:    55
VLAN Tag IDs:   888
Invalid VLAN ID Packets: 0
Port VLAN ID:    91
VLAN Tag IDs:  None

 

“So looks ok. Thanks for your help”.

 

 

 

 

A colleague of mine contacted me during the week to ask how one could determine if a POWER7 system was capable of Active Memory Expansion. I thought I'd share my response with everyone who follows my blog.

 

You can check if the system is capable of providing Active Memory Expansion (AME), from the HMC command line using the lssyscfg command, as shown here:
 
hscroot@hmc1:~> lssyscfg -r sys -m Server-8233-E8B-SN1000 -F active_mem_expansion_capable
1

Alternatively you can view the capabilities of the managed system from the HMC GUI, under System properties/Capabilities, as shown in the following image. 

 
 

Once you’ve concluded that your POWER7 system is AME capable, the next step is to check if AME is enabled or disabled for an LPAR. This is easily done from the AIX command line with the lparstat and/or amepat commands, as shown in the following example:
 
;Running lparstat and amepat on a LPAR with AME disabled.

 

$ lparstat -i| grep Exp
Target Memory Expansion Factor             : -
Target Memory Expansion Size               : -
 
$ amepat | grep Expan
WARNING: Running in no modeling mode.
Active Memory Expansion        : Disabled

 

;Running lparstat and amepat on a LPAR with AME enabled.

 

$ lparstat -i | grep -i exp

Memory Mode                                : Dedicated-Expanded

Target Memory Expansion Factor             : 1.00

Target Memory Expansion Size               : 6144 MB

 

$ amepat | grep -i exp

WARNING: Running in no modeling mode.

Active Memory Expansion        : Enabled

Target Expanded Memory Size    : 6.00 GB

Target Memory Expansion factor : 1.00
 
Note that I deliberately ran these commands as a non-root user to highlight the fact that you don’t need root access to ascertain whether or not AME is active on an LPAR.

 

By the way, did you know that AME is now supported in SAP production environments running on AIX and POWER7? Both SAP application servers, as well as database servers running DB2 LUW are supported. Unfortunately there’s no support for LPARs with Oracle RDBMS at this stage. Thanks to George Manousos from IBM Australia for providing me with this update.

Please refer to the following SAP note.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/1464605%28Power7AME%29.pdf

 

The SAP note also states:

 

Specific tests have been successfully executed with SAP ERP 6.0 (ABAP)

workload. The results of those test can be found in the following whitepaper:

 

ftp://ftp.software.ibm.com/common/ssi/sa/wh/n/pow03038usen/POW03038USEN.PDF

 

Restrictions

As AME is not recommended with large page support, AME will disable AIX

64KB pages by default.

 

AME monitoring capability in CCMS

Monitoring capabilities are available with saposcol v12.46 for more details see SAP note 710975.

 

 

It looks as though SAP have been quick to support AME on AIX/POWER7. This is a great benefit to SAP customers running with PowerVM on the IBM POWER platform. The following screenshot shows the SAP CCMS with AME statistics being reported.

 

PowerVM Integration into SAP CCMS Monitoring expanded to include Virtualized Memory Metrics

The SAP AIX porting team continues in further improving the integration of PowerVM into SAP system monitoring. While in the past already most processor and AIX virtualization metrics could be monitored via CCMS, now POWER7 Active Memory Expansion (AME)  has been included.  Customers will find a new memory section in the respective CCMS-panel as depicted in this screen shot.

For those who are not familiar with what AME actually is and how it can benefit you, I suggest you take a look at the IBM AME Wiki site first:

 

http://www.ibm.com/developerworks/wikis/display/WikiPtype/IBM+Active+Memory+Expansion

 

And watch Nigel Griffiths AME movie:

 

http://www.ibm.com/developerworks/wikis/display/wikiptype/movies#Movies-ame