MAP4040 Isolating LPAR operating system boot problems

Edit online

A checkpoint might be displayed on the operator panel for a period of time while the boot image is retrieved from the device. If the checkpoint is displayed for an extended period of time and the NVMe drive LED is not indicating any activity, there might be a problem loading the boot image from the device.

MAP4040 Section-1

Edit online

Procedure

Look at the service action event error log in the HMC Service Focal Point. Perform the actions necessary to resolve any open entries that affect devices in the boot path of the partition. Then, try to reboot the partition. Does the partition reboot successfully?

Yes, this ends the procedure.
No, go to MAP4040 Section-2.

MAP4040 Section-2

Edit online

About this task

This section shows you how to use the SMS menus to display the status of the partition boot devices (NVMe drives).

Procedure

Log on to the Management Console (HMC) with the CE user ID and password (default serv1cece).
Use the Service Utilities to quiesce and shut down the failing partition:
1. From the navigation area, click Storage Facility Management > storage facility > SF image.
2. From the right work area, select the affected LPAR.
3. From the bottom Task area, click Service Utilities > Change/Show LPAR State. The LPAR Server Control window opens.
4. Click Quiesce LPAR. Click Yes to confirm.
5. Wait 10 minutes until the Quiesce started! window opens. Click OK.
6. On the Server Control window, click Refresh to see the current status.
7. When the quiesce is complete, click Shutdown LPAR. Click Yes to confirm.
8. Wait 10 minutes until the Shutdown started window opens. Click OK.
9. On the Server Control window, click Refresh to see the current status.
10. Wait until the Operational State is Deactivated.
11. Leave the Server Control window open.
12. From the bottom Task area, click Service Utilities > Set No-rsStart.
13. When the Set No-rsStart Successful window opens, click OK.
Open a Terminal window:
1. From the navigation area, click Storage Facility Management > storage facility > Server View > server.
2. From the right work area, select the failing partition (the state of the failing partition is Not Activated).
3. From the bottom Task area, click Console Window > Open Terminal Window.

Activate the partition and interrupt access through the SMS menus.

On the Server Control window, click Activate LPAR. Click Yes to confirm.

Quickly return to the Terminal Window. When the following screen is displayed, type 1 and press Enter.

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM 
                                                                                
          1 = SMS Menu                          5 = Default Boot List           
          8 = Open Firmware Prompt              6 = Stored Boot List            
                                                                                
                                                                                
     memory      keyboard     network     scsi

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

          1 = SMS Menu                          5 = Default Boot List
          8 = Open Firmware Prompt              6 = Stored Boot List


     Memory      Keyboard     Network     SCSI     Speaker

The SMS menu is displayed. Type 1 to choose English. The SMS Main menu is displayed.
The SMS menu is displayed. Type 1 and press Enter to Select Language.
The SMS language menu is displayed. Type 1 and press Enter to choose English. The SMS Main menu is displayed.

Use the SMS menus to display the list of available boot devices:
1. On the SMS Main menu, type 5 and press return to choose Boot Options.
2. On the next menu, type 1 and press return to choose Select Install/Boot Device
3. On the next menu, type 7 and press return to choose List all Devices.
Use the SMS menus to display the list of available boot devices:
1. On the SMS Main menu, type 5 and press Enter to choose Select Boot Options.
2. On the next menu, type 1 and press Enter to choose Select Install/Boot Device
3. On the next menu, type 7 and press Enter to choose List all Devices.

Review the list of devices that are found. In some cases, you might need to enter N (Next page of list) to see the entire list of available boot devices. Also see the figures and tables in this section, which show the NVMe drive location codes for the different DS8000 models.

Figure 1. Front drive and service indicator locations (Model A01)

Figure 2. Front drive and service indicator (Model A05)

Table 1. NVMe drive locations (Models A0x). This table presents information on NVMe drive locations for Models A0x.
Model	Storage Facility Image (SFI)	Partition name	Location code
Model	Storage Facility Image (SFI)	Partition name	Primary boot drive	Secondary boot drive
A01	First	SFxxxxxxx01 (in upper CEC)	U78D3.001.xxxxxxx-P2-C1	U78D3.001.xxxxxxx-P2-C2
A01	First	SFxxxxxxx11 (in lower CEC)	U78D3.001.xxxxxxx-P2-C1	U78D3.001.xxxxxxx-P2-C2
A05	First	SFxxxxxxx01 (in upper CEC)	U78D2.001.xxxxxxx-P2-C1	U78D3.001.xxxxxxx-P2-C2
A05	First	SFxxxxxxx11 (in lower CEC)	U78D2.001.xxxxxxx-P2-C1	U78D3.001.xxxxxxx-P2-C2

Are two NVMe drives listed for the failing partition?
- Yes, go to MAP4040 Section-3
- No, go to MAP4040 Section-4.

MAP4040 Section-3

Edit online

About this task

This section isolates a boot problem where both NVMe drives are listed on the boot list.

Procedure

Remove the NVMe drive that is in position 1 on the boot list and retry the operation.
See Exchange the CEC enclosure NVMe drive (A01) or Exchange the CEC enclosure NVMe drive (A05) to remove the NVMe drive in position 1 on the boot list. Do not install a new drive at this time.
1. Type x and press enter to exit the SMS menus and begin loading the 2107/5341 operating system. Keep the terminal window open.
Did the boot problem occur again?
- Yes, go to the next step.
- No, it appears that the unplugged NVMe drive was faulty. Record the code level and installation date from the terminal window login herald.
  Go to MAP4040 Section-5.
Reinstall the removed NVMe drive. See Exchange the CEC enclosure NVMe drive (A01) or Exchange the CEC enclosure NVMe drive (A05).
Remove the NVMe drive in position 2 on the boot list.
Shutdown and activate the partition to retest it.
1. Return to the Server Control window. Click Shutdown. Click Yes to confirm.
2. Wait 10 minutes until the Shutdown started window opens. Click OK.
3. On the Server Control window, click Refresh to view the current status.
4. Wait until the Operational State is Deactivated.
5. Click Activate LPAR. Click Yes to confirm.
Did the boot problem occur again?
- Yes, go to the next step.
- No, it appears that the unplugged NVMe drive was faulty. Record the code level and installation date from the terminal window login herald.
  Go to MAP4040 Section-5.
There are two likely causes of this problem. Contact your next level of support for guidance to either:
- Reload both NVMe drives (see MAP4020).
  Remember that the No-rsStart function has been set for the affected LPAR and must be reset before resuming the LPAR.
- Replace the NVMe drive backplane assembly (Storage Facility Management > storage facility > Exchange Parts).
- After the repair is complete, go to MAP4040 Section-4, step 8.

MAP4040 Section-4

Edit online

About this task

This section isolates a boot problem where only one NVMe drive is listed on the boot list.

Procedure

Use Table 1 to identify the location for the NVMe drive that is not listed.
Remove the NVMe drive that was identified in step 1 and retry the operation.
1. See Exchange the CEC enclosure NVMe drive (A01) or Exchange the CEC enclosure NVMe drive (A05) to remove the NVMe drive identified in step 1. Do not install a new drive at this time.
2. Type x and press Enter to exit the SMS menus and begin loading the 2107/5341 operating system. Keep the terminal window open.
Did the boot problem occur again?
- Yes, go to the next step.
- No, it appears that the unplugged NVMe drive was faulty. Record the code level and installation date from the terminal window login herald.
  Go to MAP4040 Section-5.
Reinstall the removed NVMe drive. See Exchange the CEC enclosure NVMe drive (A01) or Exchange the CEC enclosure NVMe drive (A05).
Remove the NVMe drive that was listed on the boot list.
Shutdown and activate the partition to retest it.
1. Return to the Server Control window. Click Shutdown. Click Yes to confirm.
2. Wait 10 minutes until the Shutdown started window opens. Click OK.
3. On the Server Control window, click Refresh to view the current status.
4. Wait until the Operational State is Deactivated.
5. Click Activate LPAR. Click Yes to confirm.
Did the boot problem occur again?
- Yes, go to the next step.
- No, it appears that the unplugged NVMe drive was faulty. Record the code level and install date from the terminal window login herald.
  Go to MAP4040 Section-5.
Replace the disk drive backplane assembly (Storage Facility Management > storage facility > Exchange Parts).
1. After the repair is complete, use step 6 to retest.
Did the boot problem occur again?
- Yes, contact your next level of support.
- No, it appears that the NVMe drive backplane assembly was faulty. Record the code level and install date from the terminal window login herald. (See an example in MAP4040 Section-5 , step 2.)
  Go to MAP4040 Section-5.

MAP4040 Section-5

Edit online

About this task

This section cleans up after recovering the LPAR operating system boot problem.

Procedure

Determine the installed SF LIC level:

From the legacy service interface:
1. From the navigation area, click Updates.
2. From the right work area, select the storage facility.
3. From the bottom Task area, click Display Storage Facility Code Levels.
From the storage management GUI:
1. Click Service > Install Firmware > Display Storage Facility Levels
Examples: SFI Code Levels VRMF: 7.10.0.635 locationCode: 9009-42G*0662F52-V1 VRMF: 7.10.0.635 locationCode: 9009-42G*0662F22-V1

Examples: CDA Install History Package: SEA.sfi, MTMS: 9009-42G*0662F22-V1 Date: 2024/06/27-13:54, Bundle VRMF: 10.0.156.0, Package Level: 7.10.0.635, Mode: CCL
Compare the code level (code EC) recorded from the login herald with the installed level (SFI level VRMF, SEA.sfi package level) obtained in step 1.
Login herald example:
```
IBM System Storage Enterprise Storage Server (TM)
    2107 Model A05             SN 75-YZ581            Server 1  SF75YZ580ESS01
    OS Level 7.3.2.111         Code EC 7.10.0.635     Installed on: Jul 11 2024
```
Do the VRMF/EC levels match?
1. Yes, go to the next step.
2. No, it appears that the LPAR was booted from the wrong multibos image.
  
  Contact your next level of support to run the "recovery" section of the DS8000 Field Tip entitled "AIX boots old code level (SFG, MES, Rack disc, FSP repair, Model Conversion)." Inform the next level of support that the affected LPAR is already quiesced and that the No-rsStart function has been set.
Use the Service Utilities to reset No-rsStart and resume the affected LPAR. (This step should not be necessary if the "recovery" section was done in step 2.)
1. From the navigation area, click Storage Facility Management > storage facility > SF image.
2. From the right work area, select the affected LPAR.
3. From the bottom Task area, click Service Utilities > Reset No-rsStart.
4. When the Reset No-rsStart Successful window opens, click OK.
5. From the bottom Task area, click Service Utilities > Change/Show LPAR State. The LPAR Server Control window opens.
6. Click Resume LPAR. Click Yes to confirm. Allow the resume to complete.
Was the NVMe drive backplane assembly replaced?
- Yes, this completes this procedure.
- No, it appears that the unplugged NVMe drive was faulty. A serviceable event should be created that lists the NVMe drive to remove as a FRU.
  Repair the serviceable event to replace the faulty NVMe drive. If a serviceable event is not found, the NVMe drive can be replaced using the Exchange Parts menu (Storage Facility Management > storage facility > Exchange Parts).