IBM®
Skip to main content
    Country/region [select]      Terms of use
 
 
    
     Home      Products      Services & solutions      Support & downloads      My account     
 
developerworks > My developerWorks >  Dashboard > Linux for Power Architecture > ... > Performance Tuning > Re-building a RHEL 5 kernel for Power > Information > Page Comparison
developerWorks
Log In   View a printable version of the current page.
Overview Connect Spaces Forums Wikis
Re-building a RHEL 5 kernel for Power
Browse space Browse Space    
Version 32 by billburos
on Oct 09, 2009 14:37.
<< Changes from 31 to 32
compared with
Current by billburos
on Nov 17, 2009 16:20.
 
Key
These lines were removed. This word was removed.
These lines were added. This word was added.

View page history


There are 6 changes. View first change.

 {tip:title=For discussions or questions...}
 To start a discussion or get a question answered, consider posting on the [Linux for Power Architecture forum|http://www.ibm.com/developerworks/forums/forum.jspa?forumID=375].
  
 Additional Linux on Power performance information is available on the [Performance page| http://www.ibm.com/developerworks/wikis/display/LinuxP/Performance]
 {tip}
  
 *Contents*
 {toc}
  
 \\
 Many thanks to Peter Wong for providing this article.
  
 \\
 h1. Introduction
  
 {note:title=RHEL 5.3}
 For RHEL 5.3, if you build with 4KB pages, you'll get a kernel build break. Reported on the [forum|http://www.ibm.com/developerworks/forums/thread.jspa?threadID=268511]. This wiki page has been updated with the steps to remove the 64K-only patch that causes the problem when building for 4K pages. A defect has been opened.
 {note}
  
 
 \\
We are often asked to build a RHEL 5 kernel (e.g., RHEL 5.2) on Power systems for evaluating a new kernel patch or investigating a performance issue. It is actually fairly easy to build the distro kernel if you are careful with the details.
  h1. Introduction
  
Here we will present the simple steps for re-building the ppc64 RHEL 5.2 kernel as delivered by the distro. In addition, we will provide instructions to build a kernel with 4KB pages and also with a kernel patch. These steps should also be applicable to re-building the ppc64 RHEL 5.1 kernel, of course you'll need the correct RHEL 5.1 kernel src.rpm file.
  \\
 We are often asked to build a RHEL 5 kernel (e.g., RHEL 5.2, 5.3, 5.4) on Power systems for evaluating a new kernel patch or investigating a performance issue. It is actually fairly easy to build the distro kernel if you are careful with the details.
  
Here we will present the simple steps for re-building the ppc64 RHEL 5.x kernel as delivered by the distro. In addition, we will provide instructions to build a kernel with 4KB pages and also with a kernel patch.
  
 * This page is targeted at technical individuals who are running RHEL 5 on a Power system (ppc64 in the Linux community). These days this generally means the Power5 and Power6 chip architecture. Re-building kernel.org kernels is done a lot by programmers in the community and is a fairly straight-forward process, but the process of re-building the actual shipping distro kernel remains a mystery to many. So the intent here is we want to be fairly specific - what pieces do you need, what steps do you follow, and how to you tell it worked. We're not convinced these steps literally produce the "exact same kernel" as provided by the distro, but for our purposes, this is adequate.
  
 But first, we have a very serious warning. Re-building and re-booting a distro kernel is, in concept and in practice, a fairly easy thing to do. But it is also very easy to mess your system up with a single simple typo, with the practical consequences that you may need to re-install your system. Being unable to re-boot your shared system where someone else has data that wasn't backed up is a "most unpleasant" feeling.
 {warning:title=Be Wary}
 [Danger, Will Robinson|http://en.wikipedia.org/wiki/Danger,_Will_Robinson]
  
 *First*. We strongly urge you to practice these steps on a victim system, where you have no problems what-so-ever of completely re-installing the system. A "victim system" means there is nothing of importance on the system. No important data. No critical users or applications.
  
 *Second*. These steps are provided simply as a step-by-step guide of how several of us re-build the kernels in the labs on freshly installed victim systems. We most-certainly are not the official source for the right steps, nor do we claim this will work for everyone, so you should not expect any level of support from us on this. See the "First" point above.
  
 *Third*. This should be intuitive, but amazingly-enough often isn't clearly understood. Therefore, we'll say it here explicitly. If you re-build your own kernel and you use that kernel, you have invalidated any hope for "official support" from your service provider while using that kernel. See the "First" point above.
 {warning}
  
 \\
 h1. Find and install the RHEL 5.x kernel source
  
 For re-building RHEL 5.2, you'll want kernel-2.6.18-92.el5.src.rpm. It is on the source CD that comes with your CD set. You will not find it on your standard CDs.
  
 For re-building RHEL 5.3, you'll want kernel-2.6.18-128.el5.src.rpm file.
  
 for re-building RHEL 5.4, you'll want kernel-2.6.18-164.el5.src.rpm
  
 As an example, we copied the source DVD for the RHEL 5.x release to our system, mounted it, and installed the rpm.
  
 \\
 h2. For RHEL 5.2
 {noformat}
  
 copy the Source DVD iso to local system - it's a big file... 2,843,054,080 bytes..
  
 # mkdir /dvd
 # mount -o loop RHEL5.2-Server-20080430.0-SRPMS-DVD.iso /dvd
 # cd /dvd
 # find . -name kernel*.src.rpm
 ./SRPMS/kernel-2.6.18-92.el5.src.rpm
 # cd SRPMS
 # rpm -i kernel-2.6.18-92.el5.src.rpm
  (you may see lots of warning messages which can be ignored..)
 warning: user brewbuilder does not exist - using root
 warning: group brewbuilder does not exist - using root
  
 # cd /usr/src/redhat/SPECS
 # ls -l
 total 384
 -rw-r--r-- 1 root root 389007 Apr 29 2008 kernel-2.6.spec
 {noformat}
  
 \\
 h2. For RHEL 5.3
 {noformat}
 # mkdir /dvd
 # mount -o loop RHEL5.3-Server-20090106.0-SRPMS-DVD.iso /dvd
 # cd /dvd
 # find . -name kernel*
 ./SRPMS/kernel-2.6.18-128.el5.src.rpm
 # cd SRPMS
 # rpm -i kernel-2.6.18-128.el5.src.rpm
  (you may see lots of warning messages which can be ignored..)
 warning: user mockbuild does not exist - using root
 warning: group mockbuild does not exist - using root
  
 # cd /usr/src/redhat/SPECS/
 # ls -l
 total 544
 -rw-r--r-- 1 root root 551064 Dec 17 2008 kernel-2.6.spec
 {noformat}
  
 \\
 h2. For RHEL 5.4
 {noformat}
 # mkdir /dvd
 # mount -o loop RHEL5.4-Server-20090819.0-SRPMS-DVD.iso /dvd
 # cd /dvd
 # find . -name kernel*
 ./SRPMS/kernel-2.6.18-164.el5.src.rpm
 # cd SRPMS
 # rpm -i kernel-2.6.18-164.el5.src.rpm
  (you may see lots of warning messages which can be ignored..)
 warning: user mockbuild does not exist - using root
 warning: group mockbuild does not exist - using root
  
 # cd /usr/src/redhat/SPECS/
 # ls -l
 total 544
 -rw-r--r-- 1 root root 551064 Dec 17 2008 kernel-2.6.spec
 {noformat}
  
  
  
 To determine whether you have installed the source RPM, look under /usr/src/redhat/SPECS. If there is nothing there, then you probably have not installed the correct RPM yet.
  
  
  
 \\
 h1. Build the kernel as-is
  
 You might have to install unifdef-1.171-5.fc6.ppc.rpm which is needed for the build.
  
 The build instructions are as follows. This works for both RHEL 5.2 and RHEL 5.3.
  
  
 First, let's just re-build the kernel. If you're feeling brave and want to jump directly to the 4KB page build, scroll down to that section.
  
  
 \\
 h2. Edit the kernel-2.6.spec file
  
 {noformat}
 cd /usr/src/redhat/SPECS/
  
 vi kernel-2.6.spec
  
  (Search for...)
 #% define buildid
  
  (Change the commented out line *and* remove the blank space between % and define)
 #% define buildid
  
  (to something informative - in this case - the normal 64K kernel)
  (the "." is good to have.. it becomes a part of the kernel name)
 %define buildid .64K
  
 <save and file>
 {noformat}
  
  
 \\
 h2. rpmbuild
  
 Then we build.
 \\
  
 {noformat}
  
 cd /usr/src/redhat
 rpmbuild -ba --target ppc64 SPECS/kernel-2.6.spec
  
 {noformat}
  
  * "--target ppc64" is used to indicate that you want to build a ppc64 kernel
  * the rpmbuild process will take a while to complete
  
 Note: During the compilation, gpg (GNU Privacy Guard) may want some system activity to generate security related keys, which causes the build process to pause.
  
 * If this happens, you can open up another session and run "dd" command to generate some extra work or anything that generate system activities. The paused build process should automatically resume after the request from gpg is satisfied.
  
  
 Here's an example of the error:
 {noformat}
 + gpg --homedir . --batch --gen-key /usr/src/redhat/SOURCES/genkey
 gpg: WARNING: unsafe permissions on homedir `.'
 ++++++++++..+++++++++++++++.++++++++++.+++++.+++++++++++++++.++++++++++++++++++++.
 +++++.++++++++++++++++++++.+++++.++++++++++..+++++..+++++++++++++++!++++++++++++++++++++.
 ++++++++++.++++++++++..+++++++++++++++++++++++++++++++++++++++++++++++++++++++.++++++++++.
 +++++++++++++++.+++++!.++++++++++++++++++++.+++++++++++++++.+++++++++++++++++++++++++.
 +++++++++++++++..+++++.+++++.+++++.++++++++++.+++++...+++++.++++++++++.+++++!++++++++++++++++++++.
 .++++++++++.+++++++++++++++.+++++++++++++++++++++++++
  
 Not enough random bytes available. Please do some other work to give
 the OS a chance to collect more entropy! (Need 274 more bytes)
 {noformat}
  
 Admit'ably, this is a little strange. Here's what we have found that works... we create several background tasks that simply copying a disk image into /dev/null (the bit bucket). These are fairly long running.. which gives the paused rpmbuild a chance to re-start itself.
  
 {noformat}
 <in another window>
  
 # df
 Filesystem 1K-blocks Used Available Use% Mounted on
 /dev/sda3 20315788 6217952 13049196 33% /
 /dev/sda2 101082 26198 69665 28% /boot
  
 # dd if=/dev/sda3 of=/dev/null &
 # dd if /dev/sda3 of=/dev/null &
  
 <repeating the dd until the build process resumes.. there's got to be a smarter approach... >
  
 <you can kill the dd jobs once the build resumes.. ie: killall dd>
  
 {noformat}
  
  
  
 \\
 h2. Build completes
 rpmbuild is the clever program that applies all of the patches to generate the overall kernel source, builds the kernels, modules and packages all the resulting files in an RPM for installation.
  
 The rpmbuild step will result in these final messages:
 {noformat}
  
 <lots and lots of compilation messages...>
  
 Wrote: /usr/src/redhat/SRPMS/kernel-2.6.18-92.el5.64K.src.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-devel-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-headers-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debug-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debug-debuginfo-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debug-devel-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-kdump-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-kdump-devel-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debuginfo-common-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debuginfo-2.6.18-92.el5.64K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-kdump-debuginfo-2.6.18-92.el5.64K.ppc64.rpm
 {noformat}
  
 {info:title=Can these be done in parallel?}
 It would be most cool if those 12 rpm files could all be done simultaneously. It appears that rpmbuild does not support that - and it could be a shared working directory is used.
 {info}
  
  
 \\
 h2. Install the newly built kernel
  
 The instructions are as follows:
  
 *For RHEL 5.2*
 {noformat}
 # cd /usr/src/packages/RPMS/ppc64
 # rpm -i kernel-2.6.18-92.el5.64K.ppc64.rpm
  
 {noformat}
  
 *For RHEL 5.3*
 {noformat}
 # cd /usr/src/packages/RPMS/ppc64
 # rpm -i kernel-2.6.18-128.el5.64K.ppc64.rpm
  
 {noformat}
  
 *For RHEL 5.4*
 {noformat}
 # cd /usr/src/packages/RPMS/ppc64
 # rpm -i kernel-2.6.18-164.el5.64K.ppc64.rpm
  
 {noformat}
  
  
  
 h2. Check /etc/yaboot.conf
 These steps install all the pieces in the right place. An example /etc/yaboot.conf file is updated as follows:
  
  
 *RHEL 5.2 example*
  
 {noformat}
 boot=/dev/sda1
 init-message=Welcome to Red Hat Enterprise Linux Server!\nHit <TAB> for boot options
  
 partition=2
 timeout=80
 install=/usr/lib/yaboot/yaboot
 delay=5
 enablecdboot
 enableofboot
 enablenetboot
 nonvram
 fstype=raw
 default=2.6.18-92.el5.6
  
 image=/vmlinuz-2.6.18-92.el5.64K
  label=2.6.18-92.el5.6
  read-only
  initrd=/initrd-2.6.18-92.el5.64K.img
  root=/dev/VolGroup00/LogVol00
  append="console=hvc0 rhgb quiet"
  
 image=/vmlinuz-2.6.18-92.el5
  label=linux
  read-only
  initrd=/initrd-2.6.18-92.el5.img
  root=/dev/VolGroup00/LogVol00
  append="console=hvc0 rhgb quiet"
 {noformat}
  
 The variable "default" is set to "2.6.18-92.el5.6" for the new kernel. We strongly recommend that you keep the boot entry for the RHEL 5.2 kernel around, in case the newly built kernel is having some boot issues.
  
 \\
 h2. Confirm the boot
  
 Indeed you must confirm that it boots correctly.
  
 You can verify the base page size by the following command:
 {noformat}
 getconf PAGESIZE
 65536
 {noformat}
 Page size of 65536 bytes clearly indicates that we are running on 64KB pages.
  
 \\
 h1. Build a kernel with the 4KB support
  
 After you have completed the above steps successfully, we have a good base for building other kernels. Let's build a 4KB kernel.
  
 Let's follow the instructions below.
 {noformat}
 cd /usr/src/redhat/SPECS
 vi kernel-2.6.spec
  
  <find the buildid line.. make sure it is uncommented.. >
  <the "." is good to have.. it becomes a part of the kernel name>
 %define buildid .4K
  <save and file>
  
 cd /usr/src/redhat/SOURCES
  
 vi kernel-2.6.18-ppc64.config
  
  Change CONFIG_PPC_64K_PAGES=y
  To CONFIG_PPC_64K_PAGES=n
  <save and file>
  
 Do the same (turn off 64K_PAGES) to
  kernel-2.6.18-ppc64-debug.config
  kernel-2.6.18-ppc64-kdump.config
  
  
 cd /usr/src/redhat
 rpmbuild -ba --target ppc64 --without kabichk SPECS/kernel-2.6.spec
  
 {noformat}
 "--without kabichk" is used to disable ABI compatibility check - which - you need to do because you just changed something that affects the compatibility.
  * Without specifying this flag, the kernel won't build by design.
  
 On RHEL 5.2, this should succeed. On RHEL 5.3, see the next section since you likely will see a build break.
  
 For RHEL 5.2: we will see the following at the end of the build:
 {noformat}
 Wrote: /usr/src/redhat/SRPMS/kernel-2.6.18-92.el5.4K.src.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-devel-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-headers-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debug-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debug-debuginfo-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debug-devel-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-kdump-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-kdump-devel-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debuginfo-common-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-debuginfo-2.6.18-92.el5.4K.ppc64.rpm
 Wrote: /usr/src/redhat/RPMS/ppc64/kernel-kdump-debuginfo-2.6.18-92.el5.4K.ppc64.rpm
 {noformat}
  
  
 \\
 ----
 h2. Error on RHEL 5.3 for 4KB pages
  
 A patch went into RHEL 5.3 on ppc64 to support the Lx86 product when x86 applications are being run on the Power system.
  
 This patch breaks the 4KB build on RHEL 5.3. You will see an error:
  
 {noformat}
 In file included from arch/powerpc/mm/mem.c:35:
 include/asm/pgalloc.h: In function 'pgd_free':
 include/asm/pgalloc.h:48: error: implicit declaration of function 'subpage_prot_free'
 make[1]: *** [arch/powerpc/mm/mem.o] Error 1
 make[1]: *** Waiting for unfinished jobs....
 make: *** [arch/powerpc/mm] Error 2
 make: *** Waiting for unfinished jobs....
 In file included from include/asm/tlb.h:19,
  from arch/powerpc/platforms/pseries/lpar.c:36:
 include/asm/pgalloc.h: In function 'pgd_free':
 include/asm/pgalloc.h:48: error: implicit declaration of function 'subpage_prot_free'
  
 {noformat}
  
 This is easy to address.
  
 Edit the SPECS/kernel-2.6.spec file.
  
 Search for and comment out Patch 23256.
  
 {noformat}
  
 # %patch23256 -p1
  
 {noformat}
  
 and re-build...
  
  
 {noformat}
  
 cd /usr/src/redhat
 rpmbuild -ba --target ppc64 --without kabichk SPECS/kernel-2.6.spec
  
 {noformat}
  
  
  
  
  
 \\
 h2. Install the 4KB kernel
  
 *For RHEL 5.2*
 {noformat}
 cd /usr/src/redhat/RPMS/ppc64
 rpm -i ppc64/kernel-2.6.18-92.el5.4K.ppc64.rpm
 {noformat}
  
 *For RHEL 5.3*
 {noformat}
 cd /usr/src/redhat/RPMS/ppc64
 rpm -i ppc64/kernel-2.6.18-128.el5.4K.ppc64.rpm
 {noformat}
  
  
 If you did the 64KB example above, you may not be able to install both the 4KB and the 64KB kernel. You would see this message in the RHEL 5.2 example.
 {noformat}
 # rpm -i kernel-2.6.18-92.el5.4K.ppc64.rpm
  package kernel-2.6.18-92.el5.64K (which is newer than kernel-2.6.18-92.el5.4K) is already installed
 {noformat}
  
 If you get this, simply uninstall the 64K kernel rpm, and try again.
  
 {noformat}
 # rpm -e kernel-2.6.18-92.el5.64K
 # rpm -i kernel-2.6.18-92.el5.4K.ppc64.rpm
 {noformat}
  
 Obviously, on RHEL 5.3 the kernel level is slightly different.
  
  
 The /etc/yaboot.conf will be modified accordingly.
 {noformat}
 boot=/dev/sda1
 init-message=Welcome to Red Hat Enterprise Linux Server!\nHit <TAB> for boot options
  
 partition=2
 timeout=80
 install=/usr/lib/yaboot/yaboot
 delay=5
 enablecdboot
 enableofboot
 enablenetboot
 nonvram
 fstype=raw
 default=2.6.18-92.el5.4
  
 image=/vmlinuz-2.6.18-92.el5.4K
  label=2.6.18-92.el5.4
  read-only
  initrd=/initrd-2.6.18-92.el5.4K.img
  root=/dev/VolGroup00/LogVol00
  append="console=hvc0 rhgb quiet"
  
 image=/vmlinuz-2.6.18-92.el5
  label=linux
  read-only
  initrd=/initrd-2.6.18-92.el5.img
  root=/dev/VolGroup00/LogVol00
  append="console=hvc0 rhgb quiet"
 {noformat}
 &nbsp;
 And reboot.
  
 You can verify the base page size by the following command:
 {noformat}
 getconf PAGESIZE
 4096
 {noformat}
 The page size clearly indicates that we are on 4KB pages.
  
  
  
  
  
  
  
  
 \\
 h1. Apply a kernel patch
  
 Let's follow the instructions below.
 {noformat}
 cd /usr/src/redhat/SPECS
 vi kernel-2.6.spec
  
 Modify buildid as specified above.
  
 Search for Patch99990 which is almost at the end of all patches.
  
 Assume that the patch right before Patch99990 is Patch12345.
  
 Add "Patch12346: abc.patch" where abc.patch is your patch.
  
 Search for %patch12345 -p1.
  
 Add %patch12346 -p1.
  
 <save and file>
  
 Add abc.patch to /usr/src/redhat/SOURCES.
  
 Run rpmbuild as specified above.
 {noformat}
  
 \\
 h1. Using the re-built RPM on other RHEL 5.2 systems
  
 It is indeed desirable to install the built RPM on other RHEL 5.2 systems, but you need to be very careful.
  
 Note that RHEL 5.2 systems are different in many ways, e.g., the setup of the volume manager.
  
 Once the RPM is installed, follow the instructions below.
 {noformat}
 cd /boot
  
 mkinitrd initrd-2.6.18-92.el5.64K.img 2.6.18-92.el5.64K
 (Generate a fresh initrd image for booting purposes.)
 {noformat}
 And then reboot.
  
 \\
 h1. In summary
  
 This quick report takes you through the practical steps of rebuilding a RHEL 5.2 kernel for Power systems. This exercise is intended simply as a way to see how the RHEL 5.2 kernel is built, installed, and booted to.

 
    About IBM      Privacy      Contact