IBM®
Skip to main content
    Country/region [select]      Terms of use
 
 
    
     Home      Products      Services & solutions      Support & downloads      My account     
 
developerworks > My developerWorks >  Dashboard > Linux for Power Architecture > ... > Performance Tuning > Re-building a SLES 10 kernel for Power > Information > Page Comparison
developerWorks
Log In   View a printable version of the current page.
Overview Connect Spaces Forums Wikis
Re-building a SLES 10 kernel for Power
Browse space Browse Space    
Version 13 by wburos
on Aug 25, 2009 09:57.

<< Changes from 12 to 13
compared with
Current by wburos
on Aug 25, 2009 10:38.
(show comment)
 
Key
These lines were removed. This word was removed.
These lines were added. This word was added.

View page history


There are 3 changes. View first change.

 {tip:title=For discussions or questions...}
 To start a discussion or get a question answered, consider posting on the [Linux for Power Architecture forum|http://www.ibm.com/developerworks/forums/forum.jspa?forumID=375].
  
 Additional Linux on Power performance information is available on the [Performance page| http://www.ibm.com/developerworks/wikis/display/LinuxP/Performance]
 {tip}
  
 For re-building the SLES 11 kernel, see this [related SLES 11 page|https://www.ibm.com/developerworks/wikis/display/LinuxP/Re-building+a+SLES+11+kernel+for+Power].
  
 \\
 ----
 *Contents*
 {toc:minLevel=2}
  
 We are often asked how to re-build the SLES 10 kernel on Power systems, in particular to "try" the 64KB page support which Power systems can support. It turns out that while the SLES 10 kernel (and subsequent service packs) does not have the important hardware enabling kernel patches for 64KB support, it mis-leadingly includes the kernel build flag since the 2.6.16 kernel had the flags at the time.
  
 On this page we show it's actually fairly easy to re-build the distro kernel if you're careful with the details.
  
 Here we'll present the simple steps for re-building the ppc64 SLES 10 sp1 (and sp2) kernel as delivered by the distribution, essentially just by flipping the CONFIG_PPC_64K_PAGES build flag to "y". 'Course, there are some extra steps that are necessary.
  
 This example *_does not_* provide the "complete" 64KB support. The SLES 10 base does not have all of the enabling kernel features which came later in 2.6.18. So this page is just an exercise in re-building a distro kernel.
  
 * In this example, your re-built SLES 10 kernel will support _software_ 64KB page sizes (as seen by the user application), but the kernel needs additional kernel patches which enable the underlying 64KB _hardware_ pages. The patches are not provided in the kernel source. The steps below can be helpful to confirm whether your application code functions correctly in a 64KB page environment, but without the underlying 64KB hardware page size support, you will not see the normal performance improvements that can be achieved with 64KB support.
  
 * We first re-build a plain vanilla kernel and verify that it boots correctly. Then we make a small change, re-build the kernel again, and boot to the new kernel.
  
 \\
 ----
 h2. First things first.
  
 But first, a warning.
  
 Re-building and re-booting a distro kernel is conceptually, and in practice, a fairly easy thing to do. But it is also very easy to mess up your system with a single simple typo, with the practical consequences that you may need to *_reinstall_* your system. Being unable to re-boot a shared system where someone else has data that wasn't backed up is a "most unpleasant" feeling.
  
 {warning:title=Be Wary}
  
 *First*. We strongly urge you to practice these steps on a victim system, where you have no problems what-so-ever of completely reinstalling the system. A "victim system" means there is nothing of importance on the system. No important data. No critical users or applications. No key work-in-progress.
  
 *Second*. These steps are provided simply as a step-by-step guide of how several of us re-built the kernel in the labs on freshly installed victim systems. We most-certainly are not the official source for the right steps, nor do we claim this will work for everyone, and you should not expect any level of support from us on this. See the "First" point above.
  
 *Third*. This should be intuitive, but amazingly-enough often isn't clearly understood. So we'll say it here explicitly. If you re-build your own kernel and you use that kernel, you have invalidated any hope for "official support" from your service provider while using that kernel. See the "First" point above.
  
 *Fourth*. For this example in particular (which turns on limited 64KB page support), the "whole" of SLES 10 has naturally never been extensively tested with 64KB pages, and some of the related software pieces and hardware I/O drivers do not have the right patches to correctly support 64KB pages. See all of the points above.
  
 Finally, we recommend upgrading to SLES 11 when 64KB pages are needed.
  
 {warning}
  
 \\
 ----
 h2. SLES 10 - find the kernel-source "src.rpm"
  
 For SLES 10 service pack 1, you'll want kernel-source-2.6.16.46-0.12.src.rpm
  
 For SLES 10 service pack 2, you'll want kernel-source-2.6.16.60-0.21.src.rpm
  
 You'll need the \*.src.rpm file. Not the kernel-source-2.6.16.xx-0.xx.rpm file. The associated \*.src.rpm file. And it is on one of the extra CD's that come with your CD set. You will not find it on your standard CDs.
  
 This rpm cleverly doesn't show up as installed if you do a rpm -qa | grep kernel-source. To tell whether you have installed the package, look under /usr/src/packages/SPECS. If there's nothing there, then you have not installed the correct rpm.
  
 \\
 ----
 h2. Install the kernel-source "src.rpm"
  
 {noformat}
 sles10sp1: rpm -i kernel-source-2.6.16.46-0.12.src.rpm
 sles10sp2: rpm -i kernel-source-2.6.16.60.0.21.src.rpm
  
 cd /usr/src/packages/SOURCES
 cp kernel-ppc64.spec /usr/src/packages/SPECS
 cd /usr/src/packages/SPECS
  
 vi kernel-ppc64.spec
  change Release: 0.12
  to Release: 0.12.001
  <save and file>
 {noformat}
  
 Adding the .001 suffix makes it much easier to keep track of the next steps here...
  
 \\
 ----
 h2. rpmbuild -ba kernel-ppc64.spec
  
 rpmbuild is the clever step that packages all of the patches that constitute the "whole" of the kernel.
  
 For some reason, there's a variable "jobs" which isn't set, so we add this to the .spec file near the top.
  By default, rpmbuild will build things serially. When you're on a large SMP system with many cores, this is annoying.
  
{noformat}
  In the "spec" file, if the variable "jobs" is set, this will parallelize the builds.
  
 Add the $define jobs line after the existing %define lines at the top of the file
  
 {noformat}
 vi kernel-ppc64.spec
  
 Name: kernel-ppc64
 %define build_kdump %([ ppc64 != kdump ] ; echo $?)
 %define build_xen %(case ppc64 in (xen*) echo 1;; (*) echo 0;; esac)
 %define build_um %([ ppc64 != um ] ; echo $?)
 %define jobs %(cat /proc/cpuinfo | grep processor | wc -l)
  <save and file>
  
 rpmbuild -ba kernel-ppc64.spec
 {noformat}
  
 The rpmbuild compiles all of the kernel, drivers, and modules.
  
 The rpmbuild step will results in these final messages:
  
 SLES 10 sp1:
 {noformat}
 Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.46-0.12.001.nosrc.rpm
 Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.46-0.12.001.ppc64.rpm
 {noformat}
  
 SLES 10 sp2:
 {noformat}
 Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.60-0.21.001.nosrc.rpm
 Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.60-0.21.001.ppc64.rpm
 {noformat}
  
  
  
 \\
 ----
 h2. Install the newly built kernel.
  
 So then we can install the new kernel rpm.
  
 SLES 10 sp1:
 {noformat}
 cd /usr/src/packages/RPMS/ppc64
 rpm -i kernel-ppc64-2.6.16.46-0.12.001.ppc64.rpm
 {noformat}
  
 SLES 10 sp2:
 {noformat}
 cd /usr/src/packages/RPMS/ppc64
 rpm -i kernel-ppc64-2.6.16.60-0.21.001.ppc64.rpm
 {noformat}
  
  
 This step installs all of the correct pieces in all of the right places. And also updates /etc/lilo.conf
  
 {noformat}
 # Modified by YaST2. Last modification on Tue Apr 1 09:24:37 CDT 2008
 activate
 timeout = 80
 boot = /dev/sdb1
 default = linux
  
 image = /boot/vmlinux-2.6.16.46-0.12.001-ppc64
  label = 2.6.16.46-0.12.001-ppc64
  initrd = /boot/initrd-2.6.16.46-0.12.001-ppc64
  root = /dev/disk/by-id/scsi-35000c5000639376f-part2
  
 image = /boot/vmlinux-2.6.16.46-0.12-ppc64
  initrd = /boot/initrd-2.6.16.46-0.12-ppc64
 ###Don't change this comment - YaST2 identifier: Original name: linux###
  label = linux
  root = /dev/disk/by-id/scsi-35000c5000639376f-part
 {noformat}
  
 So when you reboot, one of the choices will now be "2.6.16.46-0.12.001-ppc64" (or 2.6.16.60-0.21.001-ppc64 for SLES10sp2).
  
 We recommend that you reboot to this kernel and confirm that it boots correctly.
  
 We recommend that you do NOT change the default boot kernel, leaving the test kernel as an alternative choice.
  
  
 \\
 ----
 h2. Specify 64KB support in the kernel CONFIG file
  
 You have to have completed the above steps for this to work.
  
 So first we update the base config file being used. There could be a more clever way of doing this, but this was an easy step which worked within the rpmbuild process.
  
 {noformat}
 cd /usr/src/packages/SOURCES
 cp config.tar.bz2 config.tar.bz2.orig
 tar -jxf config.tar.bz2
 cd config/powerpc
 cp ppc64 ppc64.orig
 vi ppc64
  Change CONFIG_FORCE_MAX_ZONEORDER=13
  to CONFIG_FORCE_MAX_ZONEORDER=9
  
  Change # CONFIG_PPC_64K_PAGES is not set
  to CONFIG_PPC_64K_PAGES=y
  <save and file>
  
 cd ../../
 tar -cf config.tar config/
 bzip2 config.tar
 {noformat}
  
 By changing the kernel option for 64KB memory page support, this changes the kABI which the rpmbuild enforces. If you were to re-build now with rpmbuild, the build would fail due to the kABI tolerance failure.
  
 So, to get around that, edit the kernel-ppc64.spec file and change the tolerance level.
  
 {noformat}
 cd /usr/src/packages/SPECS
 vi kernel-ppc64.spec
  Change %define tolerate_kabi_changes 6
  To %define tolerate_kabi_changes 32
  <save and file>
 {noformat}
  
 Then, to keep this new kernel separate from the kernel we just built above, we'll rename this to 002.
  
 {noformat}
 vi kernel-ppc64.spec
  change Release: 0.12.001
  to Release: 0.12.002
  <save and file>
  
 rpmbuild -ba kernel-ppc64.spec
 {noformat}
  
 This should succeed.
  
 SLES10sp1:
 {noformat}
 Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.46-0.12.002.nosrc.rpm
 Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.46-0.12.002.ppc64.rpm
 {noformat}
  
 SLES10sp2:
 {noformat}
 Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.16.60-0.21.002.nosrc.rpm
 Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.16.60-0.21.002.ppc64.rpm
 {noformat}
  
  
 We install the latest "002" kernel.
  
 SLES 10sp1:
 {noformat}
 cd /usr/src/packages/RPMS/ppc64
 rpm -i kernel-ppc64-2.6.16.46-0.12.002.ppc64.rpm
 {noformat}
  
 SLES 10sp2:
 {noformat}
 cd /usr/src/packages/RPMS/ppc64
 rpm -i kernel-ppc64-2.6.16.60-0.21.002.ppc64.rpm
 {noformat}
  
  
 And reboot.
  
 At the boot> prompt from the console, you select this new kernel.
  
 To verify you have 64KB pages, you use the intuitive /usr/bin/time command. Note the almost last line which has the Page size. This is the software page size.
  
 {noformat}
 # /usr/bin/time --verbose sleep 0
  Command being timed: "sleep 0"
  User time (seconds): 0.00
  System time (seconds): 0.00
  Percent of CPU this job got: 20%
  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
  Average shared text size (kbytes): 0
  Average unshared data size (kbytes): 0
  Average stack size (kbytes): 0
  Average total size (kbytes): 0
  Maximum resident set size (kbytes): 0
  Average resident set size (kbytes): 0
  Major (requiring I/O) page faults: 0
  Minor (reclaiming a frame) page faults: 66
  Voluntary context switches: 1
  Involuntary context switches: 1
  Swaps: 0
  File system inputs: 0
  File system outputs: 0
  Socket messages sent: 0
  Socket messages received: 0
  Signals delivered: 0
  Page size (bytes): 65536
  Exit status: 0
 {noformat}
  
 To do this with one-line output... just pipe the output to grep. This query tells you the "software" page size as the kernel presents the address space to the application.
  
 {noformat}
  /usr/bin/time --verbose sleep 0 2>&1 | grep Page
  Page size (bytes): 65536
 {noformat}
  
 Under the covers of the kernel, the kernel also manages the interface to the hardware. To see what the hardware page size is:
 {noformat}
 grep Page /var/log/dmesg
 Page orders: linear mapping = 24, others = 12
 {noformat}
  
 The "12" is in the context of 2 to the 12th, which is 4096, or 4KB pages. Hmmm. That's interesting. It "should" be 16. So in our example here, we've re-built the kernel to support software 64KB pages, but something is missing in this kernel to actually support the 64KB hardware pages.
  
 \\
 ----
 h2. What's missing
  
 To fully support 64KB hardware pages, SLES 10 needs a series of enabling kernel patches and features that emerged in 2.6.18 kernel base. This article isn't intended to cover the patches needed to fully enable 64KB pages, so we stop here.
  
 \\
 ----
 h2. In summary
  
 This quick report takes you through the practical steps of rebuilding a SLES 10 sp1 kernel for Power systems. We have not yet taken you through the steps of adding your own patches. This exercise is intended simply as a way to see how the Linux kernel is built, installed, and booted to.
  
  
  

 
    About IBM      Privacy      Contact