Here we take an FAQ approach to the performance related pages. This approach is new for this year as various answers are re-vamped into a new format.

• For general quick links, see Power Quick Links
• For the index of available pages, see Performance.
• To ask a question, see the Linux for Power architecture forum

Table of Contents: Frequently Asked Questions

Controlling threads and memory

How do I run my Linpack threads - one on single core

For an example, see Linpack on POWER7

Can I dynamically change the number of 16MB pages?

Yes. We generally recommend setting to zero (clears out pages tied to a particular memory numa node), and then request the desired number of pages.

Be sure to confirm the number of pages. The kernel will only allocate the number of pages possible, which may not be the full amount requested.

echo 0 > /proc/sys/vm/nr_hugepages
echo 1000 > /proc/sys/vm/nr_hugepages
cat /proc/sys/vm/nr_hugepages

Can I dynamically set the "dscr" settings on the cores on my POWER7 system?

Can I dynamically change the SMT setting of the cores on my POWER7 system?

Can I dynamically see the current and actual CPU frequency?

Yes.

Use the latest ppc64_cpu command from the powerpc-utils package on sourceforge.

On POWER7 systems, the CPUs are pre-numbered, and are toggled offline/online as needed to control the SMT mode. This is a run-time dynamic setting.

First, be sure your operating system supports (and exploits) POWER7.

# cat /proc/cpuinfo | grep -m 1 cpu
cpu     : POWER7 (architected), altivec supported

If it says POWER7, you're in POWER7 mode.

Can I boot to SMT=2 on POWER7?

Alas, in SLES 11, it doesn't appear to be supported on the boot command itself. The default is SMT=on (or SMT=4). Adding smt-enabled=off or smt-enabled=on to the kernel boot command does work. A defect has been opened.

The latest ppc64_cpu command from powerpc-utils can be used at boot-up time to set SMT appropriately by adding the command to the boot.local file - as in the example below.

We strongly recommend adding the breadcrumb message to the boot.local file. We have found that we have forgotten that file was modified, and have been confused when we specify smt-enabled=on on the kernel boot command, and the system still booted in SMT=2 mode.

For SLES 11 - modify /etc/init.d/boot.local

echo "/etc/init.d/boot.local: ppc64_cpu --smt=2"
ppc64_cpu --smt=2
ppc64_cpu --smt

Later, you can grep /var/log/boot.msg

 # grep smt /var/log/boot.msg 
<5>Kernel command line: root=/dev/sdb1 console=hvc0
/etc/init.d/boot.local: ppc64_cpu --smt=2

or view /var/log/boot.msg

System Boot Control: Running /etc/init.d/boot.local
/etc/init.d/boot.local: ppc64_cpu --smt=2
SMT=2
done

Can I boot to SMT=off on POWER7?

Yes. Add smt-enabled=off to the append string of the boot command in /etc/lilo.conf on SLES 11.

POWER7

Does SLES 11 exploit POWER7?

Yes. When you do "cat /proc/cpuinfo" it will show POWER7 CPUs. For example, SMT=4 is supported and the VSX engine is enabled.

Can Red Hat Version 5 exploit POWER7?

No, RHEL Version 5 will not "exploit" Power7.

RHEL 5.4 is technically not supported by Red Hat on POWER7, but has been observed to "run just fine" for normal Power6 compatibility work.

RHEL 5.5 is officially supported, but runs in "Compatibility mode" and will appear as a POWER6 system. The kernel in RHEL 5.5 does not have the enabling needed for POWER7 exploitation (in particular - that means SMT=4 and VSX support).

Bottom line - RHEL 5 will run on a POWER7 system, but will not fully exploit the POWER7 system. RHEL 5, with the older kernel, is not expected to ever fully exploit POWER7.

For RHEL 5 users, they might consider downloading, building, and booting a 2.6.32 mainline kernel which enables the POWER7 support. See the next section for more details. Since this mode is not officially supported, we don't recommend this for mission critical applications, but we have observed this mode works fine for basic use of a POWER7 system and assessments.

Note that RHEL 6, which is now generally available, has POWER7 support included. It is based on the 2.6.32 mainline kernel.

Building a mainline kernel for POWER7

For those who know how to build a mainline kernel, and have RHEL 5.4 or RHEL 5.5 today, and nicely have access to a POWER7 system, see building a mainline kernel for Power. With these steps, you should end up with a mainline a kernel which recognizes the POWER7 processor core, which supports up to 4 SMT hardware threads and the new VSX engine. Note, running this OS level with a mainline kernel isn't officially supported by anyone on POWER7. But if you have any questions, ask on the Linux for Power architecture forum.

Is there a summary of steps for RHEL 5.5 and POWER7 exploitation

See the RHEL 5.5 checklist for POWER7.

Is LPARCFG documented somewhere?

See lparcfg man page.

How do I change SMT on a POWER7?

First, you'll need a Distro release and kernel combination which exploits the POWER7 mode.

Then we recommend you download and build the latest powerpc-utils.

If you run just ppc64_cpu, and if you don't see this in the help text, you don't have the right version.

        ppc64_cpu --smt=X             # Set SMT state to X

This version is the out of date version.

# ppc64_cpu 
Usage:
	ppc64_cpu --smt			# Get current SMT state
	ppc64_cpu --smt={on|off}	# Set SMT state

	ppc64_cpu --dscr		# Get current DSCR setting
	ppc64_cpu --dscr=<val>		# Change DSCR setting

	ppc64_cpu --smt-snooze-delay	# Get current smt-snooze-delay setting
	ppc64_cpu --smt-snooze-delay=<val> # Change smt-snooze-delay setting

How do I code to the new VSX engine?

To start, get familiar with the Power Instruction Set Architecture document at the following link. See Chapter 7 (around page 271).

• POWER ISA V2.06 (pdf file)

POWER7 Systems

Where do I find reference information on the POWER7 systems

• Redbook: IBM BladeCenter PS700, PS701 and PS702 Technical Overview and Introduction

• Redbook: IBM Power 750 and 755 Technical Overview and Introduction

• Redbook: IBM Power 770 and 780 Technical Overview and Introduction

Advance Toolchain (AT) FAQs

Where can I download the Advance Toolchain?

There are now two recommended versions available, for SLES and RHEL.

There's a Version 2.1-x level and a newer Version 3.0 level.

For RHEL v5

• AT 2.1-1
• AT 3.0

For SLES 10

• AT 2.1-1
• AT 3.0

For SLES 11

• AT 2.1-1
• AT 3.0

For questions, see Advance Toolchain forum

Can I build the Advance Toolchain myself?

Not recommended. The source tar ball is available - but the build steps are non-trivial.

The Advance Toolchain provides the latest stabilized gcc tailored and tuned for POWER systems. The cpu-tuned libraries and additional packages are good to have. The Advance Toolchain is packaged in three easy to install rpms.

We recommend Advance Toolchain Version 2.1-1.

How do I use Advance Toolchain?

To download:

• ftp://linuxpatch.ncsa.uiuc.edu/toolchain/at/at05/redhat/RHEL5/
• ftp://linuxpatch.ncsa.uiuc.edu/toolchain/at/at05/suse

# ls -l advance*
total 347560
-rw-r--r-- 1 root root  62208140 Jan 29 09:42 advance-toolchain-devel-2.1-1.ppc64.rpm
-rw-r--r-- 1 root root  56310640 Jan 29 09:43 advance-toolchain-perf-2.1-1.ppc64.rpm
-rw-r--r-- 1 root root 237014085 Jan 29 09:43 advance-toolchain-runtime-2.1-1.ppc64.rpm

# rpm -i advance*.rpm
# export PATH=/opt/at05/bin:$PATH
# which gcc

For more details.. see How to use Advance Toolchain

How do I re-build a mainline kernel with Advance Toolchain?

Here's an example. When we build a recent mainline kernel on RHEL 5, the latest mainline requires a newer gcc version.. There's an easy example of building the mainline kernel for POWER7 systems on RHEL 5.4 with Advance Toolchain 2.1-1.

How do I compile with XLC and link with Advance Toolchain cpu-tuned libraries?

Why isn't objdump showing me the POWER7 VSX codes?

Use -Mpower7 on the objdump command to tell it to correctly parse for the VSX codes.

objdump -xd -Mpower7 <filename>

For example, here's an example of xlC -qlist output for a code sequence

19| 00D664 vspltisw 1000038C 1 VSPLTISW vs32=0 19| 00D668 addi 38601AB0 1 LI gr3=6832 
19| 00D66C addi 38007A30 1 LI gr0=31280 
19| 00D670 addi 38C07A20 1 LI gr6=31264 19| 00D674 addi 38811B40 1 AI gr4=gr1,6976 
19| 00D678 addi 39001AC0 1 LI gr8=6848 
19| 00D67C xvcpsgnd F0400787 1 LRVS vs34=vs32 19| 00D680 xvcpsgnd F0600787 1 LRVS vs35=vs32 
19| 00D684 stxvd2x 7C410799 1 VSTQD #vsSPILL8(gr1,gr0,0,offset=312 
80)=vs34 
19| 00D688 xvcpsgnd F0800787 1 LRVS vs36=vs32 
19| 00D68C xvcpsgnd F0A00787 1 LRVS vs37=vs32 19| 00D690 addi 38001AD0 1 LI gr0=6864 
19| 00D694 stxvd2x 7C011F99 1 VSTQD __60(gr1,gr3,0)=vs32

objdump (missing the Power7 directive) shows:

    d664:       10 00 03 8c     vspltisw v0,0 
    d668:       38 60 1a b0     li       r3,6832 
    d66c:       38 00 7a 30     li       r0,31280 
    d670:       38 c0 7a 20     li       r6,31264 
    d674:       38 81 1b 40     addi     r4,r1,6976 
    d678:       39 00 1a c0     li       r8,6848 
    d67c:       f0 40 07 87     .long 0xf0400787 
    d680:       f0 60 07 87     .long 0xf0600787 
    d684:       7c 41 07 99     .long 0x7c410799 
    d688:       f0 80 07 87     .long 0xf0800787 
    d68c:       f0 a0 07 87     .long 0xf0a00787 
    d690:       38 00 1a d0     li       r0,6864 

objdump -Mpower7 (from the Advance Toolchain) correctly shows

    d664:       10 00 03 8c     vspltisw v0,0
    d668:       38 60 1a b0     li      r3,6832
    d66c:       38 00 7a 30     li      r0,31280
    d670:       38 c0 7a 20     li      r6,31264
    d674:       38 81 1b 40     addi    r4,r1,6976
    d678:       39 00 1a c0     li      r8,6848
    d67c:       f0 40 07 87     xvmovdp vs34,vs32
    d680:       f0 60 07 87     xvmovdp vs35,vs32
    d684:       7c 41 07 99     stxvd2x vs34,r1,r0
    d688:       f0 80 07 87     xvmovdp vs36,vs32
    d68c:       f0 a0 07 87     xvmovdp vs37,vs32
    d690:       38 00 1a d0     li      r0,6864

Performance data

How do I gather performance information for a problem?

For gathering information, there's a script here .

What performance tools are available for Linux?

For recommended tools to use on Linux, see this performance tools page. The recommended Linux tools are available in the community.

Where is vmlinux on RHEL5?

You need to install kernel-debuginfo. Then search for "vmlinux" in /usr

# find /usr -name vmlinux
/usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux

For more details, see taking advantage of oprofile

Is ganglia telling me the right CPU utilization on my Power systems?

Not sure, but check out Michael Perzl's page on extensions for ganglia:

http://www.perzl.org/ganglia/

Is there a way of determining the number of TLB misses?

Check out taking advantage of oprofile and look at the section on leveraging hardware counters. oprofile is very flexible providing access to the hardwware counters.

Another approach is to look at the latest libhugetlbfs package on sourceforge.

• Sourceforge: libhugetlbfs
• Huge pages part 1: Introduction
• Huge pages part 2: Interfaces

How can I reduce the number of sample buffer overflows in oprofile?

See handling oprofile sample buffer overflows. The article discusses some of the features of oprofile which allow the user to tailor and tune the sizes of the various event buffers used by oprofile.

For example:

# Set the event (main) buffer size
opcontrol --buffer-size=1000000
 
# Set the buffer watershed
opcontrol --buffer-watershed=250000
 
# Set the cpu buffer size
opcontrol --cpu-buffer-size=16000

Are there any papers describing mitigating OS Jitter

• Practical experiences with OS Jitter

• OS Jitter Mitigation Techniques

How do I best leverage the new "perf" tool in Linux?

"perf" is based on the new performance events subsystem in the newer mainline kernels like 2.6.32.

• Using perf on POWER7 systems

Application questions

How do I build a 64-bit perl engine?

Under RHEL5, the default compilation mode is 32-bit. On this page, we provide an easy way to over-ride the default compilation mode.

• See building a 64-bit perl

How do I tune Linpack on POWER7?

For an example, see the advice for tuning Linpack on POWER7.

How do I tune HPC Challenge on POWER7?

For an example, see Tuning considerations for HPC Challenge.

How does Linux on Power support both 32- and 64-bit applications?

The current SLES and RHEL versions of Linux running on POWER systems are fully Biarch and support both 32- and 64-bit applications for development and run-time. Development (compilation and linking) may depend on installed availability of both 32bit and 64bit development RPMs for your distro.

While the Linux Enterprise distros fully support both 32- and 64-bit images for run-time and development, the default install may only include the default bitness (32- or 64-bit) run-time libraries, but the equivalent RPM for the other bit-ness should be available for install.

Another aspect to consider is the default setting of the gcc compiler (-m32 and -m64).

Why is the RHEL 5 default for applications 32-bit and the SLES 11 default 64-bit?

Historically, the default compilation mode for applications has been -m32 (32-bit) mode to ease the transition from previously existing 32-bit ecosystems. That 32-bit default approach continued from SLES9 to SLES10, from RHEL4 to RHEL5, and continues on the most recent Advance Toolchain 2.1 release. Of course, it is easy to over-ride the 32-bit default and build 64-bit applications.

Now that 64-bit applications are becoming more of the norm, the default compilation mode is slowly beginning to change to 64-bit (ie: -m64) moving forward. This started with SLES11. We expect the next version of Advance Toolchain will change its default to 64-bit. It is logical that the next version of RHEL would likely move to a 64-bit default.

So if you're compiling on both SLES 11 and RHEL 5, you'll observe two different defaults - SLES 11 being 64-bit default - and RHEL 5 being 32-bit. For legacy applications, this can a bit of a nuisance.

• Therefore, with gcc and the IBM XL compilers, we always recommend specifying the desired bitness in your makefiles to eliminate any ambiguity of the target mode.

When using IBM XL compilers, the controls are -q32 and -q64.

General debugging questions

How do I tell who's throwing random signals?

Use SystemTap to easily trace which PID is sending signals and which PID is receiving that signal.