How to properly prepare your system and replace a SAS or SSD disk unit with Linux on IBM Power Systems

1.1 This is an example of a JBOD (Just a Bunch of Disks) hdisk. From command line type in "iprconfig" then select 1 for "Display hardware status", The screen shot shows one "Physical Disk" on redundant SAS controllers, sda and sdb.

1.2 This is an example of a JBOD (Just a Bunch of Disks) hdisk when booting from standalone diagnostics. For info on booting standalone diagnostics see Appendix A1 then return here.

1.3 This is an example of a  failed hdisk when booting from standalone diagnostics. The disk may show up normally or not at all, even though it is defective and you are unable to boot your OS. For info on booting standalone diagnostics see Appendix A1 then return here. The only way to ensure the disk is defective and needs to be replaced is to run Advance Diagnostics and Certify on the disk unit, refer to Appendix A3 for this procedure.

1.4 If you have a failed JBOD disk unit with the operating system on it, your only option is to boot from standalone diagnostics. You can display what your current configuration is and then replicate it once you install the new disk unit.

1.5 Power off the LPAR or standalone system and physically replace the disk unit. Use Appendix A1 to boot standalone diagnostics and access the SAS RAID Manager menu then return here.

1.6 Select "List SAS Disk Array Configuration".

1.7 New disk units come from IBM formatted as a pdisk array candidate. This screen shot shows my new pdisk which is available to format to a JBOD hdisk. If for some reason your disk comes as an hdisk then you can skip to step 1.13.

1.8 Press F3 to get to the "IBM SAS Disk Array Manager" and then select "Change/Show SAS pdisk status".

1.9 Move the cursor to "Delete an Array Candidate pdisk and Format to JBOD block size" then press enter.

1.10 Move you cursor to your new pdisk and press enter.

1.11 You will receive a warning, press enter to continue.

1.12 The "Format in Progress" screen is displayed and will progress until the format is complete. Once complete press enter to continue.

1.13 If you want to see your new JBOD hdisk, back out one menu by pressing F3 and then select "List SAS Disk Array Configuration" and then select your primary SAS controller.

1.14 You can now shutdown your LPAR to re-install your operating system, scratch install, or from a backup. This procedure is now complete.

2.1 This is an example of an "Optimized" RAID0 array with a single disk with linux operating. From command line type in "iprconfig" then select 2 for "Work with disk arrays", then 1 for "Display disk array status". The screen shot shows one RAID0 array on redundant SAS controllers, sda and sdb.

2.2 This is an example of an "Optimal" RAID0 array with a single disk when booting from standalone diagnostics. For info on booting standalone diagnostics see Appendix A1 then return here.

2.3 This is an example of a RAID0 array with a failed disk when booting from standalone diagnostics. For info on booting standalone diagnostics see Appendix A1 then return here. If you want to test this disk unit go to Appendix A4 for the procedures then return here.

2.4 If you have a failed disk unit and no RAID protection,  with the operating system on it, your only option is to boot from standalone diagnostics. You can display what your current configuration is and then replicate it once you install the new disk unit.

2.5 Power off the LPAR or standalone system and physically replace the disk unit. Use Appendix A1 to boot standalone diagnostics and access the SAS RAID Manager menu then return here.

2.6 Select "List SAS Disk Array Configuration".

2.7 This screen shot shows my new pdisk which is available to create a new RAID0 array. If your display shows the previous RAID0 array that is in a broken state you need to back up a menu and select the option to delete the array.

2.8 Press F3 to get to the "IBM SAS Disk Array Manager" and then select "Create a SAS Disk Array".

2.9 Select your primary SAS controller then select your RAID level of "0" and press enter.

2.10 Select your stripe size, the "recommended" value is usually best, press enter to continue.

2.11 Move the cursor down to your new pdisk and press enter.

2.12 A summary of your selection is displayed, press enter to continue and your array will be created.

2.13 If you want to see your new array back out one menu by pressing F3 and then select "List SAS Disk Array Configuration" and you primary SAS controller.

2.14 You can now shutdown your LPAR to re-install your operating system, scratch install, or from a backup. This procedure is now complete.

3.1 Boot you system to Standalone Diagnostics. If you need assistance with booting to Standalone Diagnostics then go to Appendix A and follow that procedure then return here. Select the "Task Selection" menu by selecting 3 and pressing enter.

3.2 You will be prompted to set your terminal type, vt320 emulation is a good choice since the Function keys work.

3.3 Cursor to the bottom and select "Raid Array Manager"

3.4 Select "IBM SAS DIsk Array Manager".

3.5 Cursor to "List SAS Disk Array Configuration" and press enter, then select the "Primary" SAS adapter if you have redundant adapters.

3.6 This is a typical example of what a healthy RAID5 array looks like when it is set up for a linux image.

3.7 This is an example of a degraded RAID5 array with one failed disk unit.

3.8 Press F3 twice to back out to the "Task Selection List". Cursor up to "Hot Plug Task" and press enter.

3.9 Cursor to "SCSI and SCSI RAID Hot Plug Manager" and press enter.

3.10 Cursor to "Replace/Remove a Device Attached to an SCSI Hot Swap Enclosure Device" and press enter.

3.11 Cursor to the disk you want to replace and press enter. Note: the disk can display as populated or it could also show the pdisk# that failed. From step 3.7 it was pdisk2 that was failed, you could see that listed here as pdisk2 instead of populated. We know that pdisk0 and pdisk1 are our active disks in this "degraded" array.

3.12 The LED will be in the remove state, solid amber. Physically remove the disk and put the new one in then press enter to continue.

3.13 The array will not start to auto rebuild. At this point you can power down and start the Operating System and start the rebuild from the "iprconfig" utility or you can start the rebuild here. To start the rebuild here press F3 twice to get to the main "Tasks Selection List" and cursor down to "RAID Array Manager" and press enter.

3.14 Select " IBM SAS Disk Array Manager"

3.15 Cursor down to "Reconstruct a SAS Disk Array" and press enter.

3.16 The next step will depend on one of the 3 situations since standalone diagnostics may not configure the new disk.

• If you receive an error like the screen shot below you can elect to power down, boot the operating system and perform the reconstruct from “iprconfig”. If you choose this option skip to step 3.22
• If you receive an error message like the screen shot below, you can shutdown and boot standalone diagnostics again, repeat steps 3.1 to 3.4 then return here to step 3.17
• If you do not receive an error message and the screen is displayed with the newly replaced pdisk then skip to step 3.18

3.17 After the reboot I am now back at the " IBM SAS Disk Array Manager" main menu, select "Reconstruct a SAS Disk Array".

3.18 Select the new pdisk and press enter.

3.19 A summary screen will be displayed, press enter to continue the reconstruct. The reconstruct will start and bring you back to the main menu.

3.20 At this point the procedure is complete. You can wait for the rebuild to complete before powering down or you can power down and start the Operating System. To best extra safe it is preferable to wait for the rebuild to complete although it is not necessary. The rebuild will continue when the Operating System is started. If you choose to wait for the rebuild to complete, then you can display the progress by selecting "List SAS Disk Array Configuration" on the main menu.

3.21 For example, I rebooted to the Operating System to show how the rebuild continues. 

3.22 To start the reconstruct from the Operating System after a reboot, type "iprconfig" at the command line. Select option 3 for "Work with disk unit recovery".

4.1 This is a normally a concurrent procedure therefore there is no need to shut down the operating system. From command line type in "iprconfig" to access "IBM Power RAID Configuration Utility".

4.2 Select Option 2 to Work with Disk arrays.

4.3 Select Option 1 to Display Disk Array Status.

4.4 This first example is what you would see if there was NO disk issue and your array was optimal.

4.5 This second example is when it has been determined that you have a failed or missing disk that needs to be replaced.

Disk "0000:00:01.0/0:0:5:0         RAID 10 Array Member       Failed" needs replacing.

 Put a 1 next to the "Failed" disk and press enter to display the disk location information.

4.6 This will give you the disk serial number for confirmation and the physical disk location that you will need to record for the removal procedure. In this example it is "U78AB.001.WZSGCAZ-P3-D2".

4.7 Press 'q' three times to  back out to the main menu "IBM Power RAID Configuration Utility" and select 3 for "Work with disk unit recovery".

4.8 Select option 2 for "Concurrent remove device".

4.9 Put a 1 next to the disk location you recorded earlier and press enter. Note: This is an example and a test environment therefore in order to create the disk error I had to pull the disk, therefore it says the slot is empty.

4.10 Verify the information is correct and press "enter" to continue.

4.11 Physically pull the disk unit out of the system then press enter.

4.12 This will return you to the "Work with Disk Unit Recovery" menu. Select option 1 to "Concurrent add device".

4.13 Put a 1 by the same disk location recorded earlier that you will be inserting the new disk in.

4.14 Verify the information is correct then press enter to continue.

4.15 Insert the new disk unit the press enter to continue.

4.16 Select 5 to "Rebuild disk unit data".

4.17 Enter 1 to select the Failed RAID 10 array member to be rebuilt.

4.18 Confirm your selection and press enter to continue.

4.19 This will bring you to the "Work with Disk Unit Recovery" menu. Press "q" once to bring you to the "IBM Power RAID Configuration Utility" then press "1" to Display Hardware Status". You can see that the array is starting to rebuild at 4%. If you want to monitor the rebuild progress you can press the "r" key to refresh.

4.20 Note: The alternate way of showing the rebuild status is to select 2 "Work with disk arrays" from the main "IBM Power RAID Configuration Utility" iprconfig main menu then select 1 to "Display disk array status".

4.22 Once the rebuild is complete the status will show "Optimized". This procedure is now complete, your disk has been successfully replaced concurrently.

5.1. The following screenshot shows what a protected RAID5/6 array with NO hot spare and a failed disk unit looks like. This can be accessed by typing "iprconfig" at the command line and selecting option 2 "Work with disk arrays" and then 1 to "Display disk array status".

5.2 Press "q" twice to get back to the main menu or if you are at command line type in "iprconfig" and press enter. Select 3 for "Work With disk unit recovery".

5.3 Select Option 2 for a "Concurrent remove device"

5.4 This shows the "Failed" disk and the location should match what you recorded earlier. You select "1" beside the disk to be replaced and press enter. Note that you can use the "t" key to display different location formats. The second screen shot shows the physical disk location.

5.4 Press 1 and enter to start the disk removal procedure.

5.5 Verify the disk and press enter to continue.

5.6 Remove the select disk unit from the system. There should be an amber LED on for the disk to be removed.

5.7 Once the disk has been removed, press enter and the menu will go back to the "Work with Disk Unit Recovery" menu.

5.8 Select "Option 1" by pressing 1 and enter to access the "Concurrent add device" task.

5.9 Use the "t" to toggle and ensure that you select the correct location to add the new disk unit. See both examples of this screen shot below. Select the slot to add the disk to by putting a "1" beside the slot and pressing enter.

5.10 Verify the concurrent add and press enter to continue.

5.11 Insert the new disk and press enter to continue.

5.12 This will return you to the "Work with Disk Unit Recovery" menu.

6.1 The following two screenshots show what a protected array with a hot spare looks like. They can be accessed by typing "iprconfig" at the command line and selecting option 1 "Display hardware status". The second method is selecting 2 at the main menu, "Work with disk arrays" and then 1 to "Display disk array status".

6.2 When a disk in the array fails the hot spare will join the array and start the rebuild process. If you are quick enough after the disk fails you will be able to see the progress. Type "iprconfig" from the command line and select option 2 "Work with disk arrays" then option 1 "Display disk array status". The example below shows the status when the hot spare joins and the array starts to rebuild.

6.3 Press "r" to refresh the screen and show the progress of the rebuild status.

The failed disk is shown from the "Display Hardware Status" screen.

6.4 Once the rebuild is complete the array will be optimized from the "Display Disk Array Status" screen.

From "Display Hardware Status" you see the failed disk and the Optimized array.

6.5 To replace the disk type in "iprconfig" from the command line and then select option 3 for "Work with disk unit recovery".

6.6 Select 2 for "Concurrent remove device".

7.1 Display the status of your array and hot spare, from command line enter "iprconfig" then select 2 for "Work with Disk Units".

7.2 Select 1 to "Display disk array status".

7.3 The following screen show shows a healthy array with an active hot spare. It shows sda and sdb since there are redundant SAS controllers.

7.4 When a disk in the array fails it will look like the following screen shot when you catch it before it rebuilds. You are able to see the progress in the automatic rebuild process. Record the "PCI/SCSI Location" of the failed disk, this will be used later when replacing the disk unit. 

7.5 You can also display the failed hot spare from the 'iprconfig' main menu by selecting 1 to "Display hardware status". Make note of the "PCI/SCSI Location", it will be used later when replacing the disk unit.

Once the array completes the rebuild from the "Hot Spare" it will look like the following screen shot.

7.6 To replace the disk type in "iprconfig" from the command line and then select option 3 for "Work with disk unit recovery".

7.7 Select 2 for "Concurrent remove device".

Note: The following procedures are from an HMC managed system using a Linux full system LPAR. The procedures for a stand-alone system (none HMC managed) are the same except you will use the console to press the "1" key when POST is presented. This will gain you access to SMS. Diagnostic images can be downloaded from here and can be burned to DVD or USB media.

A1.1 The target LPAR should not be activated, deactivate it if it is. Prior to activating the LPAR, put your standalone diagnostic media in the system. Put a check mark in the Profile.

A1.2 Select the Actions tab and then "Activate".

A1.3 Expand Advanced Settings and change the Boot Mode to "System Management Services".

A1.4 You can put a check mark in the open VTERM check box if you want to open a console terminal window here. The method I prefer is to ssh into the HMC and use "vtmenu".

A1.5 Click on Finish and the LPAR will activate, once this process completes you can close the window and go to your console window.

A1.6 If you chose to use vtmenu from the HMC command line, ssh into the HMC as hscroot and type "vtmenu".

A1.7 Select the LPAR

A1.8 You will see the post menu. If you are on a standalone system this is where you would press "1" to force it to go to SMS. In this example we set the LPAR to activate to SMS.

A1.9 The console will come to the SMS main menu, select 5 for "Select Boot Options"

A1.10 Select 1 for "Select Install/Boot Device"

A1.11 Select 7 for "List all Devices".

A1.12 This will display a list of devices you can boot from. Often the network or fiber devices should up first so you have to press the "N" key to go to the next page. You will see the DVD or USB listed depending on what media you have loaded. Select the number of the diagnostic media and press enter, this process will start the IPL s.

A1.13 Select 2 for a Normal Mode Boot

A1.14 Select 1 to confirm it is okay to exit SMS

A1.15 Press 1 and enter to make this the console.

A1.16  Press Enter to continue.

A1.17 - This will bring you to the Diagnostics Main Menu. See the "Note" at the bottom, once you make a selection you will have to enter the terminal type, vt320 is a good option since the function keys work with this emulation.

A1.18 This ends a Standalone Diagnostics IPL, return to the step that brought you here.

A2.1 Type in iprconfig to get to the "IBM Power RAID Configuration Utility" then select 1 to "Display hardware status".

A2.2 Normally a new disk from IBM stock does not come this way, when a disk is moved from another system or was previously part of a RAID array it will show up like this and will need to be formatted.

A2.3 Press "e" to exit then press 3 to "Work with disk unit recovery".

A2.4 Select 3 to "Initialize and format disk".

A2.5 Select 1 and enter to "Initialize and Format" the disk, then press "c" to continue.

A2.6 The progress of the format is displayed.

A2.7 Once the format is complete it will bring to to the "Work with Disk Unit Recovery" menu. The ends the procedure, return to the step that brought you here.