For discussions or questions... To start a discussion or get a question answered, consider posting on the Linux for Power Architecture forum. Additional Linux on Power performance information is available on the Performance page

Contents

We are often asked how to re-build the SLES 10 kernel on Power systems, in particular to "try" the 64KB page support which Power systems can support. It turns out that while the SLES 10 kernel (and subsequent service packs) does not have the important hardware enabling kernel patches for 64KB support, it mis-leadingly includes the kernel build flag since the 2.6.16 kernel had the flags at the time.

On this page we show it's actually fairly easy to re-build the distro kernel if you're careful with the details.

Here we'll present the simple steps for re-building the ppc64 SLES 10 sp1 (and sp2) kernel as delivered by the distribution, essentially just by flipping the CONFIG_PPC_64K_PAGES build flag to "y". 'Course, there are some extra steps that are necessary.

This example does not provide the "complete" 64KB support. The SLES 10 base does not have all of the enabling kernel features which came later in 2.6.18. So this page is just an exercise in re-building a distro kernel.

• In this example, your re-built SLES 10 kernel will support software 64KB page sizes (as seen by the user application), but the kernel needs additional kernel patches which enable the underlying 64KB hardware pages. The patches are not provided in the kernel source. The steps below can be helpful to confirm whether your application code functions correctly in a 64KB page environment, but without the underlying 64KB hardware page size support, you will not see the normal performance improvements that can be achieved with 64KB support.

• We first re-build a plain vanilla kernel and verify that it boots correctly. Then we make a small change, re-build the kernel again, and boot to the new kernel.

First things first.

But first, a very serious warning. Re-building and re-booting a distro kernel is conceptually, and in practice, a fairly easy thing to do. But it is also very easy to mess up your system with a single simple typo, with the practical consequences that you may need to reinstall your system. Being unable to re-boot a shared system where someone else has data that wasn't backed up is a "most unpleasant" feeling.

Be Wary Danger, Will Robinson First. We strongly urge you to practice these steps on a victim system, where you have no problems what-so-ever of completely reinstalling the system. A "victim system" means there is nothing of importance on the system. No important data. No critical users or applications. No key work-in-progress. Second. These steps are provided simply as a step-by-step guide of how several of us re-built the kernel in the labs on freshly installed victim systems. We most-certainly are not the official source for the right steps, nor do we claim this will work for everyone, and you should not expect any level of support from us on this. See the "First" point above. Third. This should be intuitive, but amazingly-enough often isn't clearly understood. So we'll say it here explicitly. If you re-build your own kernel and you use that kernel, you have invalidated any hope for "official support" from your service provider while using that kernel. See the "First" point above. Fourth. For this example in particular (which tries to turns on 64KB support), the "whole" of SLES 10 has naturally never been extensively tested with 64KB pages, and some of the related software pieces and hardware I/O drivers do not have the right patches to correctly support 64KB pages. See "All" of the points above.

SLES 10 sp - the kernel-source

For SLES 10 service pack 1, you'll want kernel-source-2.6.16.46-0.12.src.rpm

For SLES 10 service pack 2, you'll want kernel-source-2.6.16.60-0.21.src.rpm

You'll need the *.src.rpm file. Not the kernel-source-2.6.16.xx-0.xx.rpm file. The associated *.src.rpm file. And it is on one of the extra CD's that come with your CD set. You will not find it on your standard CDs.

This rpm cleverly doesn't show up as installed if you do a rpm -qa | grep kernel-source. To tell whether you have installed the package, look under /usr/src/packages/SPECS. If there's nothing there, then you have not installed the correct rpm.

Install the kernel-source-*.src.rpm

sles10sp1: rpm -i kernel-source-2.6.16.46-0.12.src.rpm
sles10sp2: rpm -i kernel-source-2.6.16.60.0.21.src.rpm

cd /usr/src/packages/SOURCES
cp kernel-ppc64.spec /usr/src/packages/SPECS
cd /usr/src/packages/SPECS

vi kernel-ppc64.spec
     change  Release:    0.12    
     to      Release:    0.12.001
     <save and file>

Adding the .001 suffix makes it much easier to keep track of the next steps here...

rpmbuild -ba kernel-ppc64.spec

rpmbuild is the clever step that packages all of the patches that constitute the "whole" of the kernel.

For some reason, there's a variable "jobs" which isn't set, so we add this to the .spec file near the top.

Add the $define jobs line after the existing %define lines at the top of the file

vi kernel-ppc64.spec

Name:           kernel-ppc64
%define build_kdump %([ ppc64 != kdump ] ; echo $?)
%define build_xen %(case ppc64 in (xen*) echo 1;; (*) echo 0;; esac)
%define build_um  %([ ppc64 != um ] ; echo $?)
%define jobs %(cat /proc/cpuinfo | grep processor | wc -l)
    <save and file>

rpmbuild  -ba kernel-ppc64.spec

The rpmbuild compiles all of the kernel, drivers, and modules.

The rpmbuild step will results in these final messages:

SLES 10 sp1:

Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.46-0.12.001.nosrc.rpm
Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.46-0.12.001.ppc64.rpm

SLES 10 sp2:

Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.60-0.21.001.nosrc.rpm
Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.60-0.21.001.ppc64.rpm

Install the newly built kernel.

So then we can install the new kernel rpm.

SLES 10 sp1:

cd /usr/src/packages/RPMS/ppc64
rpm -i kernel-ppc64-2.6.16.46-0.12.001.ppc64.rpm

SLES 10 sp2:

cd /usr/src/packages/RPMS/ppc64
rpm -i kernel-ppc64-2.6.16.60-0.21.001.ppc64.rpm

This step installs all of the correct pieces in all of the right places. And also updates /etc/lilo.conf

# Modified by YaST2. Last modification on Tue Apr  1 09:24:37 CDT 2008
activate
timeout = 80
boot = /dev/sdb1
default = linux

image = /boot/vmlinux-2.6.16.46-0.12.001-ppc64
    label = 2.6.16.46-0.12.001-ppc64
    initrd = /boot/initrd-2.6.16.46-0.12.001-ppc64
    root = /dev/disk/by-id/scsi-35000c5000639376f-part2

image = /boot/vmlinux-2.6.16.46-0.12-ppc64
    initrd = /boot/initrd-2.6.16.46-0.12-ppc64
###Don't change this comment - YaST2 identifier: Original name: linux###
    label = linux
    root = /dev/disk/by-id/scsi-35000c5000639376f-part

So when you reboot, one of the choices will now be "2.6.16.46-0.12.001-ppc64" (or 2.6.16.60-0.21.001-ppc64 for SLES10sp2).

We recommend that you reboot to this kernel and confirm that it boots correctly.

We recommend that you do NOT change the default boot kernel, leaving the test kernel as an alternative choice.

Request 64KB support in the kernel CONFIG file

You have to have completed the above steps for this to work.

So first we update the base config file being used. There could be a more clever way of doing this, but this was an easy step which worked within the rpmbuild process.

cd /usr/src/packages/SOURCES
cp config.tar.bz2 config.tar.bz2.orig
tar -jxf config.tar.bz2
cd config/powerpc
cp ppc64 ppc64.orig
vi ppc64
     Change CONFIG_FORCE_MAX_ZONEORDER=13
     to     CONFIG_FORCE_MAX_ZONEORDER=9

     Change # CONFIG_PPC_64K_PAGES is not set
     to     CONFIG_PPC_64K_PAGES=y
     <save and file>

cd ../../
tar -cf config.tar config/
bzip2 config.tar

By changing the kernel option for 64KB memory page support, this changes the kABI which the rpmbuild enforces. If you were to re-build now with rpmbuild, the build would fail due to the kABI tolerance failure.

So, to get around that, edit the kernel-ppc64.spec file and change the tolerance level.

cd /usr/src/packages/SPECS
vi kernel-ppc64.spec
     Change %define tolerate_kabi_changes 6
     To     %define tolerate_kabi_changes 32
     <save and file>

Then, to keep this new kernel separate from the kernel we just built above, we'll rename this to 002.

vi kernel-ppc64.spec
     change  Release:    0.12.001
     to      Release:    0.12.002
     <save and file>

rpmbuild -ba kernel-ppc64.spec

This should succeed.

SLES10sp1:

Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.46-0.12.002.nosrc.rpm
Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.46-0.12.002.ppc64.rpm

SLES10sp2:

Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.60-0.21.002.nosrc.rpm
Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.60-0.21.002.ppc64.rpm

We install the latest "002" kernel.

SLES 10sp1:

cd /usr/src/packages/RPMS/ppc64
rpm -i kernel-ppc64-2.6.16.46-0.12.002.ppc64.rpm

SLES 10sp2:

cd /usr/src/packages/RPMS/ppc64
rpm -i kernel-ppc64-2.6.16.60-0.21.002.ppc64.rpm

And reboot.

At the boot> prompt from the console, you select this new kernel.

To verify you have 64KB pages, you use the intuitive /usr/bin/time command. Note the almost last line which has the Page size. This is the software page size.

# /usr/bin/time --verbose sleep 0
        Command being timed: "sleep 0"
        User time (seconds): 0.00
        System time (seconds): 0.00
        Percent of CPU this job got: 20%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 0
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 66
        Voluntary context switches: 1
        Involuntary context switches: 1
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 65536
        Exit status: 0

To do this with one-line output... just pipe the output to grep. This query tells you the "software" page size as the kernel presents the address space to the application.

 /usr/bin/time --verbose sleep 0 2>&1 | grep Page
        Page size (bytes): 65536

Under the covers of the kernel, the kernel also manages the interface to the hardware. To see what the hardware page size is:

grep Page /var/log/dmesg
Page orders: linear mapping = 24, others = 12

The "12" is in the context of 2 to the 12th, which is 4096, or 4KB pages. Hmmm. That's interesting. It "should" be 16. So in our example here, we've re-built the kernel to support software 64KB pages, but something is missing in this kernel to actually support the 64KB hardware pages.

What's missing

To fully support 64KB hardware pages, SLES 10 needs a series of enabling kernel patches and features that emerged in 2.6.18 kernel base. This article isn't intended to cover the patches needed to fully enable 64KB pages, so we stop here.

In summary

This quick report takes you through the practical steps of rebuilding a SLES 10 sp1 kernel for Power systems. We have not yet taken you through the steps of adding your own patches. This exercise is intended simply as a way to see how the Linux kernel is built, installed, and booted to.