Introduction

This article explains step-by-step how to set up software-based redundant array of independent disks (RAID)-1 under RHEL 3.0 on an IBM F50 (7026-6F1) that had 2 x 36.4GB internal SCSI disks. The instructions are specific to this setup, but they can easily be modified to work with a different pSeries model and Linux distribution.

Back to top

Assumptions

The article assumes you're familiar with the installation of Linux on pSeries and the RedHat Enterprise Linux 3.0 installation program. It is also assumed you have the skills to use system tools such as fdisk, rpm, vi, dd, and so on. Knowledge of kernel compilation and the generic Linux boot up process will be helpful.

We assume that you understand the limitation of a software based RAID solution compared to a hardware RAID solution, and do not intend to use it for production data.

Back to top

Required hardware and software

• A Linux certified IBM pSeries system with at least two internal disk drives, over which a software RAID is to be set up
• Red Hat Enterprise Linux AS 3.0 for pSeries (kernel version 2.4.21-4) CD ROM Media

On the pSeries system use the SMS menu to set up the boot order as:
Diskette -> CDROM->HDD1 ->HDD2

Back to top

Procedure

Use the following steps to set up disk mirroring using software-based RAID.

1. Start RHEL 3.0 installation
2. Create disk partitions on both the disk drives using the fdisk utility
3. Create software RAID devices using the Disk Druid utility
4. Select kernel development packages and complete the installation
5. Compile kernel to create a zImage.initrd file
6. Modify the partitions tag on the disk drives
7. Backup the partition tables of the disk drives
8. Define and create software RAID device for /boot
9. Write the zImage.initrd file to the PReP Boot partitions on all the disk drives
10. Test the setup

Step 1. Start RHEL 3.0 installation

1. Power on the pSeries system and insert the RHEL 3.0 installation CD (CD #1) into the CD drive
2. When the SMS menu is displayed, verify that the boot list is set per the boot order mentioned previously
3. Set the install device to be the CDROM
4. Exit the SMS menu and let the system boot off the RHEL 3.0 installation CD
5. When the Installer prompt (boot:) is displayed, type linux to start the GUI or type linux text to start the text based installation
6. Step through the installation program as you typically would until the Disk Partitioning Setup screen is displayed
7. At the Disk Partitioning Setup screen, switch to the shell prompt using ctrl+alt+F2 (if you selected the GUI mode) or ctrl+F2 (text mode)

After the RHEL 3.0 installation program starts up, configure the language, keyboard, mouse settings, and so on. When the Disk Partitioning screen is displayed, switch to the shell prompt in order to manually create partitions on the disk.

Step 2. Create partitions on both disk drives using fdisk

1. Start the fdisk utility on the first disk by typing fdisk /dev/sda at the shell prompt
2. Create the following partitions on the first disk. When done, save the changes and quit fdisk.

   sda1 of size 8Mb with tag ‘41’ sda2 of size 100Mb with tag '83' sda3 of size 1024Mb with tag 'fd' sda4 of size 4096Mb or more with tag 'fd'

   
   
3. Start the fdisk utility on the second disk by typing fdisk /dev/sdb at the shell prompt
4. Create the following partitions on the first disk. When done, save the changes and quit fdisk.

   sdb1 of size 8Mb with tag '83' sdb2 of size 100Mb with tag '83' sdb3 of size 1024Mb with tag 'fd' sdb4 of size 4096Mb with tag 'fd'

   
   

By manually creating the partition as above, the RAID partitions when created will be matching on the drives and not out of order. If you just create partitions and then let Disk Druid do whatever it wants, you end up with them all out of order, which can cause a big headache when replacing drives and initiating rebuilds on partitions. We have identical partition order and sizes on both the disks. Also note the tags used. Tag '41' is used to specify the 'PreP' boot partition.

In our setup we created only the primary partitions. You may create sda4, sdb4 as extended partitions and then create more logical partitions (sda5, sdb5, and so on) if required. The key point, if the logical partitions are to be on software RAID-1, is that you need to tag the partition type for all the logical partitions as 'fd'. Don't tag the extended partition as 'fd'.

Step 3. Create software RAID devices using the Disk Druid utility

1. Switch back to the Disk Partitioning Setup screen by pressing Alt+F7 (GUI) or Alt+F1(Text)
2. Select the Disk Druid option. Answer yes if prompted to "format all swap partitions"
3. When the Partitioning screen is displayed, be sure that all the disk partitions created are displayed
4. If the screen displays any default RAID device (/dev/md0 or /dev/md1), select and then delete them
5. Select the partition /dev/sda2 and edit the settings so the mount point is set to /boot, and select to have it formatted as ext3. When done, return to the Partitioning Screen.
6. Use the RAID button to create new software RAID devices
7. Create a new RAID device /dev/md0 with mount point as /, file system type ext3, raid device md0, raid level raid 1, raid members sda4, sdb4, spares as 0, and choose to format partition
8. Create a new raid device /dev/md1 with file system type swap, raid device md1, raid level raid 1, raid members sda3, sdb3, spares as 0, and choose to format partition
9. Ensure that RAID device /dev/md0 is of type ext3 with mount point as / and RAID device /dev/md1 is of type swap. In the text mode installation, the devices may get interchanged and this should be corrected.
10. Verify that the partition /dev/sdb1 is NOT listed as a swap partition. If it is, then edit it and change the type to ext3 with no mount point. Leave all other partitions on /dev/sdb unchanged.
11. Move on to the next screen of the installation. Ignore the warning if displayed about smaller than recommended /boot partition, and format warnings messages and so on.

After we've created the disk partitions, we use the Disk Druid utility to define the mount points and to set up the software RAID devices. The RHEL 3.0 installer currently doesn't allow the /boot to be on a software RAID device, so we have to use /dev/sda2 to mount /boot. We also create RAID device /dev/md0 to mount the /, the system root file system, and /dev/md1 for use as the swap device. If you have additional partitions tagged for use as software RAID, set them up. Pay special attention to the RAID level and RAID member selection while creating addition RAID devices.

It is important to ensure that / is on RAID Device 0 (/dev/md0) and RAID device 1 (/dev/md1) is of type swap. In the text mode installation, the devices may get interchanged. To correct this we had to delete and recreate the RAID devices, changing the order in which the devices are created. For example, first create the RAID device for swap then for /. If that doesn't work, delete both the RAID devices and then create a RAID device for / followed by swap.

Step 4. Select kernel development packages and complete the installation

1. Continue with the installation to set hostname, networking, and so on until the Package Selection is displayed
2. At the Package Selection screen, choose Customize software selection
3. Select the list of software to be installed. Ensure that the following group are also selected:
   • Development Tools
   • Kernel Development Tools
   • Legacy Software Development
   • Compatibility Arch Development Support
4. After the software selection is completed, continue with installation

We will be using a customized kernel for the setup, which requires kernel development tools and associated software that we select here, and continue with the installation.

Once the installation is completed, the system will reboot. Sometimes after rebooting the system may stop at the yaboot prompt (boot:) with an error message about missing or invalid device. In such a case, just type linux and press Enter. The error message may be left over from an earlier run of yaboot or an installer glitch.

At this point you should have a working RHEL 3.0 installed and running on your pSeries system.

Step 5. Compile kernel to generate a zImage.initrd file

1. Logon to the system as the root user
2. Change to /usr/src/linux-2.4 directory to start kernel compilation

   # cd /usr/src/linux-2.4

   
   
3. Copy the configuration settings of the currently running kernel. In our case the running kernel was 2.4.21-4.EL

   # cp /boot/config-2.4.21-4.EL swinitrd.config

   
   
4. Edit the file Makefile to change the EXTRAVERSION variable to match the running kernel.

   EXTRAVERSION= -4.EL

   
   
5. Copy the initial ram disk (initrd) file into the kernel source tree. We used the existing intrd file present in /boot.

   # cp /boot/initrd-2.4.21-4.EL.img /usr/src/linux-2.4/arch/ppc64/boot/ ramdisk.image.gz

   
   
6. Compile the kernel to generate the single zImage.initrd file

   # cd /usr/src/linux-2.4 # make distclean # cp swinitrd.config .config # make oldconfig # make dep # make zImage.Initrd

   
   
7. Copy the newly created zImage.initrd file and the configuration file to the /boot

   # cp /usr/src/linux-2.4/arch/ppc64/boot/ zImage.initrd /boot/zImage.initrd-2.4.21-4.EL # cp /usr/src/linux-2.4/swinitrd.config /boot/ config.initrd-2.4.21-4.EL

   
   
8. Optional: Edit the /etc/yaboot.conf file and append the following lines

   image=/zImage.initrd-2.4.21-4.EL lable=initrd read-only root=/dev/md0

   
   
9. Optional: Update the boot loader so that it loads the newly created zImage.initrd on booting

   #ybin –v –b /dev/sda1 –filesystem raw

   
   
10. Optional: Verify if zImage.initrd loads correctly by rebooting the system and type initrd at the boot: prompt
11. Optional: If the zImage.initrd load fails, reboot and type linux at the boot: prompt to load the previous image

The above steps are used to create the zImage.initrd file, which combines the linux kernel (zImage) and the initial ramdisk (initrd) into a single file. We do this so that the kernel need not access the storage device to load the various device drivers required to boot up the system.

While we were able to use the initrd (which was created by the installer) already present in the system, sometimes you might have to build a custom initrd. If you create a new initrd, ensure that you include and preload all the driver modules required to boot up the system and the RAID module (raid1). Use the lsmod command to get the list of modules you need to boot up your system. As an example, we used the following command to create a custom initrd and then copied it to the kernel tree.

# mkinitrd –v –f --preload=scsi_mod --preload=sd_mod –-
preload=sr_mod 
   --preload=sym53c8xx --preload=raid1 --preload=jbd  --
preload=ext3 
      /boot/initrd.raid1-2.4.21-4.EL.img  2.4.21-4.EL
# cp /boot/initrd.raid1-2.4.21-4.EL.img /usr/src/linux-2.4/arch/ppc64
   /boot/ramdisk.image.gz

The list of modules and especially the SCSI adapter module name (sym53c8xx) will differ from system to system. The module may have dependency and have to be loaded in order, for example jbd module must be loaded before ext3 module. If the modules list is incomplete or the module load order is not correct, the system will not boot up. We recommend you make use of the optional steps to verify the zImage.initrd file before continuing.

Step 6. Modify the partition tags on the disk drives

1. Logon to the system as the root user
2. Start the fdisk utility on the first hard disk by typing fdisk /dev/sda
3. Change the partition tags for the partition /dev/sda2 to 'fd' and save the changes
4. Ignore the error message displayed (don't reboot) when you exit the fdisk utility
5. Run fdisk on the second hard disk by typing fdisk /dev/sdb
6. Change the tag of partition /dev/sdb2 to 'fd'. Change the tag of partition /dev/sdb1 to '41'.
7. Set the bootable flag for /dev/sdb1 and save the changes
8. Ignore the error message displayed (don't reboot) when you exit the fdisk utility

We use fdisk to modify the partition tags on both the disk drives so that a PreP partition is now defined on both the disk drives.

Step 7. Back up partition tables of the disk drives

1. Create a directory /raidinfo
2. Use the sfdisk command to back up the existing partition tables of the disk drives, as follows

   # sfdisk -d /dev/sda > /raidinfo/sfdisk.sda # sfdisk -d /dev/sdb > /raidinfo/sfdisk.sdb

   
   
3. Copy the file containing the partition table information to a floppy for safe keeping

Saving a copy of the partition table makes replacing drives and rebuilding them a lot easier upon failure. See the Basic administration on how partitions table can be restored on a new drives using this saved file.

Step 8. Define and create software RAID devices for /boot

1. Create a directory called /boot1, copy the content of /boot to this directory, and then umount /boot

   # mkdir /boot1 # cp -dpR /boot/* /boot1/ # umount /boot

   
   
2. Edit the file /etc/raidtab and append the following lines to define a new RAID device, then save the changes. Pay special attention to the raid-dev, raid-level, device, and raid-disk line entries below.

   raiddev /dev/md2 raid-level 1 nr-raid-disks 2 chunk-size 64k persistent-superblock 1 nr-spare-disks 0 device /dev/sda2 raid-disk 0 device /dev/sdb2 raid-disk 1

   
   
3. Initialize the newly created RAID devices /dev/md1 and /dev/md0 using the mkraid command

   # mkraid -R /dev/md2

   
   
4. Ignore any warning message about the device not being clean. Check the status for /dev/md2.

   # cat /proc/mdstat

   
   
5. Bring the newly created RAID devices online. Ignore any "already running" messages.

   # raidstop /dev/md2 # raidstart /dev/md2

   
   
6. Format the RAID device as ext3 file system using the mke2fs command

   # mke2fs -j /dev/md2

   
   
7. Edit the file /etc/fstab to remove all the existing line entries for /boot, and instead append the following

   /dev/md2 /boot ext3 defaults 1 2

   
   
8. Mount the RAID device /dev/md2 as /boot, copy back the contents from /boot1 to /boot, and remove /boot1

   # mount /boot # cp -dpR /boot1/* /boot # rm –rf /boot1

   
   
9. Edit the file /etc/sysctl.conf and append the following lines to set the minimum and maximum values for RAID rebuilding speed (KB/sec)

   dev.raid.speed_limit_min = 10000 dev.raid.speed_limit_max = 40000

   
   The values (10000, 40000) above are what we used on our system. Higher values may speed up rebuilding, but will incur higher CPU and I/O bandwidth.
10. Load the sysctl settings into the kernel

    # sysctl –p

    
    

We've now transferred the /boot onto a RAID-1 device. The / and the swap partitions were already set up under RAID-1 by the installer. Thus, in case one of the disk drive fails now, the system should continue to run since all the data is mirrored. In the following section we'll set up the system so it can boot up even when one of the disk drives has failed. Do not reboot the system at this point.

Step 9. Write the new kernel image to the PReP boot partitions of all the disks

1. Erase the existing data on the PreP partition of the second disk

   # dd if=/dev/zero of=/dev/sdb1 bs=512 count=1600

   
   
2. Write the zImage.initrd file to the PreP partition of the second disk

   # dd if=/boot/zImage.initrd-2.4.21-4.EL of=/dev/sdb1 bs=512

   
   
3. Optional: Reboot the system and change the boot order using the SMS Menu to boot off the second disk
4. Write the zImage.initrd file to the PreP partition of the first disk

   # dd if=/dev/zero of=/dev/sda1 bs=512 count=1600 # dd if=/boot/zImage.initrd-2.4.21-4.EL of=/dev/sdb1 bs=512

   
   

When you install RHEL 3.0, the boot loader (yaboot) is only installed on the PreP partition of the first disk drive (/dev/sda1 in our case), so if the first disk drive fails the system will not be able to boot up. Therefore, we need to have a bootable PreP partition on the second disk drive. The system automatically goes in drive order for both IDE and SCSI disks, and uses the first active PReP Boot partition it finds.

Ideally, if the boot loader (yaboot) shipped with RHEL 3.0 AS for pSeries was able to work with /boot on software RAID device, we would only have to re-install the boot loader on the PreP partition of both the disk drives. Since it doesn't, we opt to install the kernel to the PreP partition. To ensure that the kernel has all the drivers required to start up the system, we tag the initrd file to the end of the kernel by creating the zImage.initrd file.

Once the kernel is written to the PReP boot partition, we actually don't need to have a /boot partition at all. If ever the kernel needs to be upgraded, be sure to create a new zImage.initrd file (see section 4) and have it written to the PreP partitions (see section 9) of both the disks. The /boot partition on /dev/md2 is not used in the boot-up process anymore.

Step 10. Test the setup

1. Reboot the system. When the SMS menu is displayed, be sure the boot order is set to boot off the first disk.
2. While the Linux OS is loading, monitor the log messages for any error messages
3. Login as the user root and check if the RAID devices are correctly setup

   # mount # cat /proc/mdstat

   
   
4. Run the following command to wipe off the data on the first disk

   # dd if=/dev/zero of=/dev/sda1 bs=1024 count=8192 # dd if=/dev/zero of=/dev/sda2 bs=1024 count=8192 # dd if=/dev/zero of=/dev/sda3 bs=1024 count=8192 # dd if=/dev/zero of=/dev/sda4 bs=1024 count=8192 # dd if=/dev/zero of=/dev/sda bs=1024 count=8192

   
   (CAUTION: The above will delete all data from the first disk)
5. As no valid data is now present on the first disk, mark the RAID members on the first disk as failed

   # raidsetfaulty /dev/md2 /dev/sda2 # raidsetfaulty /dev/md1 /dev/sda3 # raidsetfaulty /dev/md0 /dev/sda4

   
   
6. Access various files to confirm that the RAID driver is now using data from the mirrored partitions
7. Reboot the system. The system should now boot off the second hard disk.
8. Login as the user root and check the RAID devices. The status will show that one of the underlying devices is missing, along with other information.
9. Next restore the "failed" first disk drive back into the system using the following

   # sfdisk /dev/sda < /raidinfo/partitions.sda # raidhotadd /dev/md2 /dev/sda2 # raidhotadd /dev/md1 /dev/sda3 # raidhotadd /dev/md0 /dev/sda4

   
   
10. Write the kernel image to the PreP partition of the first disk

    # dd if=/boot/zimage-raid.2.4.19-9.EL of=/dev/sda1 bs=512

    
    
11. Check the rebuilding status of the RAID devices

    # cat /proc/mdstat

    
    
12. Optional: After rebuilding is complete, test the setup with data on the second drive zeroed

To test the setup we wipe data off the first disk drive. Because no valid data is on the first disk, we mark all the RAID member partitions on that disk as faulty. This ensures that the driver doesn't mirror the zeroed data from the first disk to the second disk if a rebuilding is accidentally initiated. Otherwise, do an immediate reboot to make the driver aware that the first disk has failed.

Since the PreP partition on the first disk is zeroed, the system will now boot off the second disk drive, which has an active PreP partition, and proceed to mount / , /boot, and swap, which have been mirrored on the second disk. After the system boots up completely we restore the partition table on the first disk, rebuild the drive partitions, and finally write the kernel image to the PreP partition of the first disk drive.

Back to top

Basic administration

This section tells you how to: update the kernel, remove a defective drive, add a new drive to replace a failed one, and add the drive partitions to the software RAID.

Updating the kernel

Install the update rpm for the kernel, along with the update rpm for kernel sources. As part of the update, the rpm will copy over to /boot the updated kernel image, the kernel config file, a new initrd, and modules in /lib/modules and kernel sources in /usr/src/linux-2.4/. Since the system no longer boots up using the /boot, you will need to create a new zImage.initrd file for the new kernel (see section 4) and have it written to the PreP partitions (see section 9) of both the disks.

Remove a defective drive from the system

Mark all the drive partitions on the failed disk as defective using the raidsetfaulty command.

If the SCSI driver supports hot swap, get the list of SCSI devices on the system and remove the defective device.

		
# cat /proc/scsi/scsi
   Host:  scsi0 Channel: 00 Id: 00 Lun: 00
   Vendor: Seagate Model: and so on
   Host:  scsi0 Channel: 00 Id: 01 Lun: 00
   Vendor: Seagate Model: and so on
# echo "scsi remove-single-device 0 0 1 0" > /proc/scsi/scsi

If the driver doesn't support hot swap, then you'll have to power down the system to remove the failed disk drive.

Add a new drive to replace a failed drive

If the SCSI driver supports hot swap functions, then to add the device (/dev/sdb) execute the following:

# echo "scsi add-single-device" 0 0 1 0 > /proc/scsi/scsi

If the driver doesn't support hot swap, then you'll have to power down the system and add a new disk drive.

To avoid confusion with device names, we recommend that you add the new drive with the system powered off (especially if the replaced drive is the first disk) and then power it back up so the first disk appears as /dev/sda and the second as /dev/sdb.

Add the drive partitions to the software RAID

Use the sfdisk command to restore the partition table on the new disk. If the first disk was replaced, then run:

# sfdisk /dev/sda < /raidinfo/partitions.sda

Install the kernel in the PReP boot.

# dd if=/boot/zimage-raid.2.4.19-9.EL of=/dev/sda1 bs=512

Check the software RAID status maintained in the file /proc/mdstat.

# less /proc/mdstat
md2 : active raid1 sdb2[1]
nnnnn blocks [2/1] [_U]

Add the drive partition back to the array using the raidhotadd command. Assuming that drive partitions are the same as the sample setup created for the software RAID, the command to run is:

		
# raidhotadd /dev/md2 /dev/sda2
# raidhotadd /dev/md1 /dev/sda3
# raidhotadd /dev/md0 /dev/sda4

Check the rebuilding status:

		
# cat /proc/mdstat
md2 : active raid1 sda2[0] sdb2[1]
nnnnn blocks [2/2] [UU]

After rebuilding is completed, both the drives will be listed as being up. The rebuild is done in the order the raidhotadd command is given.

Resources

• The Software-RAID HOWTO at: http://www.tldp.org/HOWTO/Software-RAID-HOWTO.html
  
  
• Boot + Root + Raid + Lilo : Software Raid mini-HOWTO: at http://www.tldp.org/HOWTO/Boot+Root+Raid+LILO.html
  
  
• Software Raid and Grub HOW-TOat : http://lists.us.dell.com/pipermail/linux-poweredge/2003-July/014331.html
  
  
• Setting up RAID - Linux StepByStep at : http://www.linux-sxs.org/storage/raid_setup.html
  
  
• Re: boot problem, partition= -1 at: http://lists.penguinppc.org/yaboot-users/2004/yaboot-users-200404/msg00004.html
  
  

About the authors

Rate this article

Comments

Back to top

Table of contents

• Introduction
• Assumptions
• Required hardware and software
• Procedure
• Basic administration
• Resources
• About the authors
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into Linux on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers