IBM Support

AIX Booting Hangs at LED Code 546.

Troubleshooting


Problem

When a customer is booting his AIX server/LPAR, the booting process hangs at LED Code 546.

Cause

Such an issue most likely happens when a customer adds an adapter via DLPAR without rebooting the LPAR after adding it. Most of the customers perform the procedure below:

1- They add another FC adapter via DLPAR. (The device tree in AIX copy of Open FW tree is not marked a supporting boot function)
2- They run cfgmgr and see the new disk.
3- They add the new disk to rootvg and mirror it through LVM.
4- The bosboot command should run fine but only on the original set of disks.
5- They start to see some savebase errors after they break the mirror from the old storage.
6- The issue is only fixed if they run bosboot to create a new boot image on the new storage and savebase should work as well after that.

Note: The savebase command stores the boot image information in the Object Data Manager (ODM) as such and is closely linked with the bosboot command. Typically, savebase errors occur after a hardware migration or an alt_disk migration.

The procedure below should be followed when adding or migrating rootvg disks:
Mirroring the rootvg Volume Group for AIX
https://www.ibm.com/support/pages/mirroring-rootvg-volume-group-aix

Diagnosing The Problem

The customer should capture a debug boot for the failing-to-boot AIX server/LPAR using one of the two ways below depending on his environment:
When the debug boot is finished, the customer should be able to see the below:
+ echo rc.boot: executing savebase recovery procedures
+ 1>> /../tmp/boot_log
+ + fgrep population
+ odmshow CuDv
+ ODMDIR=/etc/objrepos
exec(/usr/bin/fgrep,population){2293842,1507376}
exec(/usr/bin/odmshow,CuDv){1769542,2293842}
X=    population:     49 objects (45 active, 4 deleted)
+ + cut -f2 -d
+ echo population: 49 objects (45 active, 4 deleted)
exec(/usr/bin/cut,-f2,-d ){2293844,1507376}
count=49
+ [ 49 -ne 0 ]
+ unset AIX_NO_SAVEBASE
+ /usr/sbin/savebase
exec(/usr/sbin/savebase){2293846,1507376}
+ [ 1 -ne 0 ]
+ loopled 0x546 SAVEBASE FAILED
exec(/usr/lib/methods/showled,0x546,SAVEBASE FAILED){2293848,1507376}
 showled

So, it seems that it was not booting due to savebase is failing.

Resolving The Problem

To fix this issue, you need to follow the action plan below:

1- Boot the server into Maintenance Mode:
https://www-01.ibm.com/support/docview.wss?uid=isg3T1013056

2- Please choose option 1 ---> 1) Access this Volume Group and start a shell
In this shell, please run:

# lslv -m hd5 -----> Check for the rootvg disk.
# rmlv hd5
# chpv -c hdisk# -----> hdisk# is the rootvg disk.
# mklv -y hd5 -t boot -a e rootvg 1 hdisk# -----> hdisk# is the rootvg disk.
--------------------------------------------------
# cd /dev
# rm ipldevice
# rm ipl_blv
# ln /dev/rhdisk# /dev/ipldevice ------> hdisk# is the rootvg disk.
# ln /dev/rhd5 /dev/ipl_blv
--------------------------------------------------
# bosboot -ad /dev/ipldevice
# bosboot -ad /dev/hdisk# ------> hdisk# is the rootvg disk.
# savebase -d /dev/hdisk# ------> hdisk# is the rootvg disk.
# bootlist -m normal hdisk# ------> hdisk# is the rootvg disk.
# bootlist -m normal -o 
# sync;sync;sync;reboot -q

Document Location

Worldwide

SUPPORT

If you require more assistance, use the following step-by-step instructions to contact IBM to open a case for software with an active and valid support contract.  

1. Document (or collect screen captures of) all symptoms, errors, and messages related to your issue.

2. Capture any logs or data relevant to the situation.

3. Contact IBM to open a case:

   -For electronic support, see the IBM Support Community:
     https://www.ibm.com/mysupport
   -If you require telephone support, see the web page:
      https://www.ibm.com/planetwide/

4. Provide a clear, concise description of the issue.

 - For guidance, see: Working with IBM AIX Support: Describing the problem.

5. If the system is accessible, collect a system snap, and upload all of the details and data for your case.

 - For guidance, see: Working with IBM AIX Support: Collecting snap data

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"Component":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
03 September 2020

UID

ibm11274302

Page Feedback