PowerLinux Performance FAQs

This page has not been liked. Updated 5/7/13, 9:58 AM by Bill_BurosTags: None


PowerLinux Architecture

Here we take an FAQ approach to the performance related pages.


Controlling threads and memory


How do I run my Linpack threads - one on single core

For an example, see Linpack on POWER7


Can I dynamically change the number of 16MB pages?

Yes. We generally recommend setting to zero (clears out pages tied to a particular memory numa node), and then request the desired number of pages.

Be sure to confirm the number of pages. The kernel will only allocate the number of pages possible, which may not be the full amount requested.

echo 0 > /proc/sys/vm/nr_hugepages
echo 1000 > /proc/sys/vm/nr_hugepages
cat /proc/sys/vm/nr_hugepages


Can I dynamically set the "dscr" settings on the cores on my POWER7 system?

Can I dynamically change the SMT setting of the cores on my POWER7 system?

Can I dynamically see the current and actual CPU frequency?


Use the latest ppc64_cpu command from the yum/zypper repo.

On POWER7 systems, the CPUs are pre-numbered, and are toggled offline/online as needed to control the SMT mode. This is a run-time dynamic setting.

First, be sure your operating system supports (and exploits) POWER7.

# cat /proc/cpuinfo | grep -m 1 cpu
cpu     : POWER7 (architected), altivec supported

The ppc64_cpu command will also dynamically check the actual frequency of the cores running. Note that the values reported by "cat /proc/cpuinfo" are statically defined by the system on boot.

# ppc64_cpu --frequency
min:	3.56 GHz (cpu 57)
max:	3.56 GHz (cpu 0)
avg:	3.56 GHz


How does Linux handle SMT modes

See Simultaneous Multi-Threading on POWER7 Processors

ppc64_cpu --smt=x simply toggles the CPUs offline/online consistently across the cores.

Linux numbers all of the CPUs sequentially across the cores.

ppc64_cpu --smt=1 will "turn off" cpu1,cpu2, cpu3 of the four CPUs associated with each core, leaving cpu0 running. 0,1,2,3 is just the example for the first core. That repeats up through the CPUs.

ppc64_cpu --smt=2 insures that cpu0, cpu1 are "on", and cpu2, cpu3 are "off" for each core.

ppc64_cpu --smt=4 insures that all cpus are "on"

SMT=on is reported when all of the hardware threads are turned on.


Can I boot to SMT=2 on POWER7?

Alas, in SLES 11, it doesn't appear to be supported on the boot command itself. The default is SMT=on (or SMT=4). Adding smt-enabled=off or smt-enabled=on to the kernel boot command does work. A defect has been opened.

The latest ppc64_cpu command from powerpc-utils can be used at boot-up time to set SMT appropriately by adding the command to the boot.local file - as in the example below.

We strongly recommend adding the breadcrumb message to the boot.local file. We have found that we have forgotten that file was modified, and have been confused when we specify smt-enabled=on on the kernel boot command, and the system still booted in SMT=2 mode.

For SLES 11 - modify /etc/init.d/boot.local

echo "/etc/init.d/boot.local: ppc64_cpu --smt=2"
ppc64_cpu --smt=2
ppc64_cpu --smt

Later, you can grep /var/log/boot.msg

 # grep smt /var/log/boot.msg 
<5>Kernel command line: root=/dev/sdb1 console=hvc0
/etc/init.d/boot.local: ppc64_cpu --smt=2

or view /var/log/boot.msg

System Boot Control: Running /etc/init.d/boot.local
/etc/init.d/boot.local: ppc64_cpu --smt=2

The same process applies to RHEL 6.    Use /etc/rc.local  and check for SMT in /var/log/boot.log

Can I boot to SMT=off on POWER7?

Yes. Add smt-enabled=off to the append string of the boot command in /etc/lilo.conf on SLES 11.




Does SLES 11 exploit POWER7?

Yes. When you do "cat /proc/cpuinfo" it will show POWER7 CPUs. For example, SMT=4 is supported and the VSX engine is enabled.


Can Red Hat Version 5 exploit POWER7?

No, RHEL Version 5 will not "exploit" Power7.

RHEL 5.4 is technically not supported by Red Hat on POWER7, but has been observed to "run just fine" for normal Power6 compatibility work.

RHEL 5.5 is officially supported, but runs in "Compatibility mode" and will appear as a POWER6 system. The kernel in RHEL 5.5 does not have the enabling needed for POWER7 exploitation (in particular - that means SMT=4 and VSX support).

Bottom line - RHEL 5 will run on a POWER7 system, but will not fully exploit the POWER7 system. RHEL 5, with the older kernel, is not expected to ever fully exploit POWER7.

For RHEL 5 users, they might consider downloading, building, and booting a 2.6.32 mainline kernel which enables the POWER7 support. See the next section for more details. Since this mode is not officially supported, we don't recommend this for mission critical applications, but we have observed this mode works fine for basic use of a POWER7 system and assessments.

Note that RHEL 6, which is now generally available, has POWER7 support included. It is based on the 2.6.32 mainline kernel.

Is LPARCFG documented somewhere?

See lparcfg man page.


How do I change SMT on a POWER7?

First, you'll need a Distro release and kernel combination which exploits the POWER7 mode.

If you run just ppc64_cpu, and if you don't see this in the help text, you don't have the right version.

        ppc64_cpu --smt=X             # Set SMT state to X

This version is the out of date version.

# ppc64_cpu 
	ppc64_cpu --smt			# Get current SMT state
	ppc64_cpu --smt={on|off}	# Set SMT state

	ppc64_cpu --dscr		# Get current DSCR setting
	ppc64_cpu --dscr=<val>		# Change DSCR setting

	ppc64_cpu --smt-snooze-delay	# Get current smt-snooze-delay setting
	ppc64_cpu --smt-snooze-delay=<val> # Change smt-snooze-delay setting


How do I code to the new VSX engine?

To start, get familiar with the Power Instruction Set Architecture document at the following link. See Chapter 7 (around page 271).


POWER7 Systems


Where do I find reference information on the POWER7 systems



How do I use Advance Toolchain?

To download:

# ls -l advance*
total 347560
-rw-r--r-- 1 root root  62208140 Jan 29 09:42 advance-toolchain-devel-2.1-1.ppc64.rpm
-rw-r--r-- 1 root root  56310640 Jan 29 09:43 advance-toolchain-perf-2.1-1.ppc64.rpm
-rw-r--r-- 1 root root 237014085 Jan 29 09:43 advance-toolchain-runtime-2.1-1.ppc64.rpm

# rpm -i advance*.rpm
# export PATH=/opt/at05/bin:$PATH
# which gcc

For more details.. see How to use Advance Toolchain

Why isn't objdump showing me the POWER7 VSX codes?

Use -Mpower7 on the objdump command to tell it to correctly parse for the VSX codes.

objdump -xd -Mpower7 <filename>

For example, here's an example of xlC -qlist output for a code sequence

19| 00D664 vspltisw 1000038C 1 VSPLTISW vs32=0 19| 00D668 addi 38601AB0 1 LI gr3=6832 
19| 00D66C addi 38007A30 1 LI gr0=31280 
19| 00D670 addi 38C07A20 1 LI gr6=31264 19| 00D674 addi 38811B40 1 AI gr4=gr1,6976 
19| 00D678 addi 39001AC0 1 LI gr8=6848 
19| 00D67C xvcpsgnd F0400787 1 LRVS vs34=vs32 19| 00D680 xvcpsgnd F0600787 1 LRVS vs35=vs32 
19| 00D684 stxvd2x 7C410799 1 VSTQD #vsSPILL8(gr1,gr0,0,offset=312 
19| 00D688 xvcpsgnd F0800787 1 LRVS vs36=vs32 
19| 00D68C xvcpsgnd F0A00787 1 LRVS vs37=vs32 19| 00D690 addi 38001AD0 1 LI gr0=6864 
19| 00D694 stxvd2x 7C011F99 1 VSTQD __60(gr1,gr3,0)=vs32

objdump (missing the Power7 directive) shows:

    d664:       10 00 03 8c     vspltisw v0,0 
    d668:       38 60 1a b0     li       r3,6832 
    d66c:       38 00 7a 30     li       r0,31280 
    d670:       38 c0 7a 20     li       r6,31264 
    d674:       38 81 1b 40     addi     r4,r1,6976 
    d678:       39 00 1a c0     li       r8,6848 
    d67c:       f0 40 07 87     .long 0xf0400787 
    d680:       f0 60 07 87     .long 0xf0600787 
    d684:       7c 41 07 99     .long 0x7c410799 
    d688:       f0 80 07 87     .long 0xf0800787 
    d68c:       f0 a0 07 87     .long 0xf0a00787 
    d690:       38 00 1a d0     li       r0,6864

objdump -Mpower7 (from the Advance Toolchain) correctly shows

    d664:       10 00 03 8c     vspltisw v0,0
    d668:       38 60 1a b0     li      r3,6832
    d66c:       38 00 7a 30     li      r0,31280
    d670:       38 c0 7a 20     li      r6,31264
    d674:       38 81 1b 40     addi    r4,r1,6976
    d678:       39 00 1a c0     li      r8,6848
    d67c:       f0 40 07 87     xvmovdp vs34,vs32
    d680:       f0 60 07 87     xvmovdp vs35,vs32
    d684:       7c 41 07 99     stxvd2x vs34,r1,r0
    d688:       f0 80 07 87     xvmovdp vs36,vs32
    d68c:       f0 a0 07 87     xvmovdp vs37,vs32
    d690:       38 00 1a d0     li      r0,6864


Performance data


How do I gather performance information for a problem?

For gathering information, there's an "lpcpu" script here.


What performance tools are available for Linux?

For recommended tools to use on Linux, see this performance tools page. The recommended Linux tools are available in the community.

Is ganglia telling me the right CPU utilization on my Power systems?

Not sure, but check out Michael Perzl's page on extensions for ganglia:



Is there a way of determining the number of TLB misses?

Check out taking advantage of oprofile and look at the section on leveraging hardware counters. oprofile is very flexible providing access to the hardwware counters.

Another approach is to look at the latest libhugetlbfs package on sourceforge.


How can I reduce the number of sample buffer overflows in oprofile?

See handling oprofile sample buffer overflows. The article discusses some of the features of oprofile which allow the user to tailor and tune the sizes of the various event buffers used by oprofile.

For example:

# Set the event (main) buffer size
opcontrol --buffer-size=1000000
# Set the buffer watershed
opcontrol --buffer-watershed=250000
# Set the cpu buffer size
opcontrol --cpu-buffer-size=16000


Are there any papers describing mitigating OS Jitter


How do I best leverage the new "perf" tool in Linux?

"perf" is based on the new performance events subsystem in the newer mainline kernels like 2.6.32.


Application questions


How do I build a 64-bit perl engine?

Under RHEL5, the default compilation mode is 32-bit. On this page, we provide an easy way to over-ride the default compilation mode.


How do I tune Linpack on POWER7?

For an example, see the advice for tuning Linpack on POWER7.


How do I tune HPC Challenge on POWER7?

For an example, see Tuning considerations for HPC Challenge.


How does Linux on Power support both 32- and 64-bit applications?

The current SLES and RHEL versions of Linux running on POWER systems are fully Biarch and support both 32- and 64-bit applications for development and run-time. Development (compilation and linking) may depend on installed availability of both 32bit and 64bit development RPMs for your distro.

While the Linux Enterprise distros fully support both 32- and 64-bit images for run-time and development, the default install may only include the default bitness (32- or 64-bit) run-time libraries, but the equivalent RPM for the other bit-ness should be available for install.

Another aspect to consider is the default setting of the gcc compiler (-m32 and -m64).


Why is the RHEL 5 default for applications 32-bit and the SLES 11 default 64-bit?

Historically, the default compilation mode for applications has been -m32 (32-bit) mode to ease the transition from previously existing 32-bit ecosystems. That 32-bit default approach continued from SLES9 to SLES10, from RHEL4 to RHEL5, and continues on the most recent Advance Toolchain 2.1 release. Of course, it is easy to over-ride the 32-bit default and build 64-bit applications.

Now that 64-bit applications are becoming more of the norm, the default compilation mode is slowly beginning to change to 64-bit (ie: -m64) moving forward. This started with SLES11. We expect the next version of Advance Toolchain will change its default to 64-bit. It is logical that the next version of RHEL would likely move to a 64-bit default.

So if you're compiling on both SLES 11 and RHEL 5, you'll observe two different defaults - SLES 11 being 64-bit default - and RHEL 5 being 32-bit. For legacy applications, this can a bit of a nuisance.

  • Therefore, with gcc and the IBM XL compilers, we always recommend specifying the desired bitness in your makefiles to eliminate any ambiguity of the target mode.

When using IBM XL compilers, the controls are -q32 and -q64.