Configuring Software Raid and LVM on Power using RHEL4U2 (Red Hat Enterprise Linux Advanced Server version 4 Update 2)

Contents

Prerequisites

We are using Red Hat Enterprise Linux Advanced Server 4 (RHEL AS4) on OpenPower 720.
It is recommended to use hardware raid adapters on all server systems. Still there are cases, where hardware raid is not applicable: for example for a scenario using two VIO servers serving each one disk to the client LPARs and using software mirroring on the client side.

Background

Using software RAID on POWER is different from using it on Intel due to the different boot loader. Power uses the Yaboot1 boot loader which is loaded by Open Firmware2 after identifying the PReP boot partition on the disk. Only disks containing the PReP boot partition can be selected in SMS (user interface to the Open Firmware). Yaboot starts and scans all the primary disk partitions searching for its own configuration file (usually /etc/yaboot.conf) and the kernel image to load. Unfortunately Yaboot can not scan software mirrored md-devices or LVM logical volumes. It also contains a build in knowledge of only some file systems: ext2 , ext3, reiserfs and fat. We are going to use separate /boot partition which will be a second primary partition on both disks and contain boot kernel, initrd and etc directory with yaboot.conf inside.

Target layout

Partitioning layout

We are going to create following partitions on our first disk:

We will create almost the same partitions on the second disk, the only difference is, that the second disk should not have a PrePBoot partition. We will change the type later.

Raid layout

We will create a raid device /dev/md0 containing /dev/sda2 and /dev/sdb2
and a raid device /dev/md1 containing /dev/sda3 and /dev/sdb3

LVM layout

We will use our raid device /dev/md0 as a physical disk and include it in the volume group /dev/volgrp01. We will create following logical volumes in this volume group.

Step by step: manual installation

Start installation

Start your installation from CD or network. Power on your system or activate your LPAR and press 1 to enter SMS menu. Press 5 to enter "Select Boot Options" section, 1 in the next screen select "Select Install or Boot device" and select the boot device you are going to use.

Leave anaconda installer

After anaconda installer starts you will get the greeting screen and stop at "Disk partioning setup" screen.

Unfortunately we can not use Autopartition (it would setup LVM without any mirroring). Also the "Disk Druid" was not usable for a proper raid configuration. We are going to leave anaconda installer for a while and partition our disks manualy.
To do that press CTRL+z and you will get a shell.

Command line partitioning

Create partitions on /dev/sda (first disk)

• Use fdisk /dev/sda command. Use "h" for help, "n" for new partition, "t" for type, "a" to set boot flag, and "w" to write your partitioning to the disk.

• Create following partitions:

   Disk /dev/sda: 18.2 GB, 18253611008 bytes
   64 heads, 32 sectors/track, 17408 cylinders
   Units = cylinders of 2048 * 512 = 1048576 bytes
  
     Device Boot      Start         End      Blocks   Id  System
   /dev/sda1               1          10       10224   41  PPC PReP Boot
   /dev/sda2              11         106       98304   fd  Linux raid autodetect
   /dev/sda3             107       17408    17717248   fd  Linux raid autodetect
  

Create partitions on /dev/sdb (second disk)

In order to create the same partioning layout on the second disk we can repeat the same steps above or use sfdisk instead fdisk.

• Run "sfdisk -d /dev/sda > /tmp/sda.out" command in order to dump the partition table in to a file.

• Run "sfdisk /dev/sdb < /tmp/sda.out" in order to creat identical partition table on the second disk.
• Set the partition type to Linux/83 for /dev/sdb1 in order to avoid yaboot confusion.

Create Raid devices

Use following command in order to create a raid-device /dev/md0 containing the partitions /dev/sda2 and /dev/sdb2 for /boot:

 mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2

and for LVM:

 mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3

Create volume group and logical volumes

In order to configure LVM enter the lvm shell by issuing '''lvm''' command.

*Create physical volume:

 lvm> pvcreate /dev/md1

*Create volume group:

 lvm> vgcreate volgrp01 /dev/md1

*Create logical volumes:

 lvm> lvcreate -L 1000 --name rootlv volgrp01
 lvm> lvcreate -L 4000 --name usrlv volgrp01
 lvm> lvcreate -L 512  --name tmplv volgrp01
 lvm> lvcreate -L 512  --name homelv volgrp01
 lvm> lvcreate -L 512  --name optlv volgrp01
 lvm> lvcreate -L 512  --name varlv volgrp01
 lvm> lvcreate -L 512  --name swap volgrp01
 lvm> quit

Get back to anaconda installer

Type "exit" on the shell prompt and you will get back to anaconda installer. Select to use Disk Druid for partitioning.
Select your logical volumes and press F3 for edit. In the next screen choose file system options and set the volumes to be formatted.

One screen back set the proper mount point for each volume:

Select /dev/md0 and set it to be formatted and mounted as /boot. After finishing continue your installation, and let the system reboot.

Post Installation tasks

If everything is gone well, your system should reboot and come up with software raid /dev/md0 and /dev/md1 and LVM on it. If your system does not boot, the most likely cause will be the size of the PrepBoot partition. Make sure it is not bigger then 10 MB. You may also need to set your boot device in SMS menu.

Your partitioning should look like this:

 [root@op720-1-client2 ~]# fdisk -l

 Disk /dev/sda: 18.2 GB, 18253611008 bytes
 64 heads, 32 sectors/track, 17408 cylinders
 Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
 /dev/sda1               1          10       10224   41  PPC PReP Boot
 /dev/sda2              11         106       98304   fd  Linux raid autodetect
 /dev/sda3             107       17408    17717248   fd  Linux raid autodetect

 Disk /dev/sdb: 18.2 GB, 18253611008 bytes
 64 heads, 32 sectors/track, 17408 cylinders
 Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
 /dev/sdb1   *           1          10       10224   41  PPC PReP Boot
 /dev/sdb2              11         106       98304   fd  Linux raid autodetect
 /dev/sdb3             107       17408    17717248   fd  Linux raid autodetect
 

• And your logical volumes:

   [root@op720-1-client2 ~]# lvscan
    ACTIVE            '/dev/volgrp01/rootlv' [1000.00 MB] inherit
    ACTIVE            '/dev/volgrp01/usrlv' [3.91 GB] inherit
    ACTIVE            '/dev/volgrp01/tmplv' [1000.00 MB] inherit
    ACTIVE            '/dev/volgrp01/optlv' [1000.00 MB] inherit
    ACTIVE            '/dev/volgrp01/varlv' [1000.00 MB] inherit
    ACTIVE            '/dev/volgrp01/swaplv' [1000.00 MB] inherit
  

• And you mount points like this:

   [root@op720-1-client2 ~]# mount
   /dev/mapper/volgrp01-rootlv on / type ext3 (rw)
   none on /proc type proc (rw)
   none on /sys type sysfs (rw)
   none on /dev/pts type devpts (rw,gid=5,mode=620)
   /dev/md0 on /boot type ext3 (rw)
   none on /dev/shm type tmpfs (rw)
   /dev/mapper/volgrp01-optlv on /opt type ext3 (rw)
   /dev/mapper/volgrp01-tmplv on /tmp type ext3 (rw)
   /dev/mapper/volgrp01-usrlv on /usr type ext3 (rw)
   /dev/mapper/volgrp01-varlv on /var type ext3 (rw)
   none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
   sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
  

Mirror PrepBoot partition

We need to have the same working PrepBoot partition on the second disk in order to be able to boot if the first one is absent or broken.

• We are going to mirror the PrepBoot partition using dd:

   dd if=/dev/sda1 of=/dev/sdb1
  

• Now we set the boot flag:

   parted /dev/sdb set 1 boot on
  

Set the boot list

We also need to set the OpenFirmware boot list to contain both disks:

 bootlist -m normal sda sdb
 bootlist -m normal -o
 sda
 sdb

References

1. Linux on Power wiki page: http://oss.gonicus.de/openpower
2. Disk mirroring using software RAID on RHEL 3.0 AS for IBM pSeries: http://www-106.ibm.com/developerworks/eserver/library/es-SW_RAID_LINUX.html

Author: Tomas Baublys

Thanks to:
Brian J. King, René Bersier, Michael Perzl, Dustin Kirkland, Martin Welk