Capturing a debug NIM boot

Your AIX Support team can be able to help you resolve certain problems without having to collect debug boot information based on their experiences with common causes of certain AIX network boot issues. However, there are times when the most common solutions do not work to solve a problem, and additional information may be required to perform a complete analysis. This document will guide you through the steps of collecting the debug boot data that your AIX Support Technician requires to resolve problems like LED 605, 608, 610, and 611, hangs among other possible issues, when attempting to boot a NIM client over the network. This topic will be covered in the following parts:

• Logging the client console's output:
  • from a stand-alone system
  • from an LPAR using PuTTY
  • from an LPAR using the NIM master
• Opening the client's console:
  • on an HMC managed LPAR
  • on an IVM managed LPAR
• Performing the Debug Boot
• Closing the client's console and submitting the log file:
  • on an HMC or IVM managed LPAR
  • Submitting the log file

 

---

Logging the client console's output

Choose one of the following methods to log the debug output, which will be generated on the client system's console. You may choose any method, but the following are the most common. If the client system is not an LPAR, you will have to use the first method described, and will have to have the appropriate equipment (hardware) to collect the console's output.

Logging the client console's output from a stand-alone system

Required equipment:

• • Serial null modem cable - The one I use has a 9-pin serial connection on each end.
  • Laptop with terminal emulation software that has logging capabilities - Mine is a Windows laptop, and I use PuTTY to connect to the serial port

Optional equipment:

• • RJ45 to DB9 serial adapter - Newer Power Systems do not have 9-pin serial connectors, they have RJ45 serial ports. In this case, you will need an adapter, if you are using a 9-pin null modem cable.
  • DB9 to USB serial adapter - Your laptop, like mine, may not have a 9-pin serial connector. I have to use an adapter to connect the DB9 null modem cable to a USB port.

You will use the above equipment to connect the S1 serial port on the client system to the serial (or USB) port on the laptop. You will then use PuTTY (or some other terminal software with logging capabilities) to capture the debug output. The steps to configure this using PuTTY are:

1. Start PuTTY. You will see the "PuTTY Configuration" dialog box.
2. On the left side, you will see a tree of Categories. Select the "Logging" category, which is listed under the "Session" category tree. You will see the "Options controlling session logging" options.
3. Under "Session logging:" you will choose the "Printable output" option.
4. Under "Log file name:" enter the file name that you want to save the console output to. This should start with your full PMR number, so that it can be easily associated with your PMR when you send it to your support technician. For example: 12345.678.000.debug_boot.log. You will also want to click the "Browse..." option and choose a location to save the file, perhaps your Windows Desktop, for easy access.
5. That takes care of the logging options, now to establish the serial connection on the laptop. In the Category tree, select the "Serial" category, which is listed under the "Connection" category tree. You will see the "Options controlling local serial lines" options.
6. Enter the correct COM port in the "Serial line to connect to" option. If you are not sure which COM port to choose, it might be helpful to use the Windows Device Manager. Look under the "Ports (COM & LPT)" device tree for the available COM ports on your laptop.
7. In the "Configure the serial line" section, the "Speed (baud)" should be either 19200 (for newer systems) or 9600 (for older systems). The "Data/Stop/Parity" should be 8/1/None, respectively. The "Flow control" should be "XON/XOFF."
8. Click the "Open" option to establish the serial connection. Once the connection is established, you may need to press the Enter key a few times. If you see non-printable characters on the screen, try using the other "Speed (baud)" setting.

Logging the client console's output from an LPAR using PuTTY

If the client system is an LPAR, it is much simpler to capture the console output. Since most people use the PuTTY telnet/SSH client, it's convenient to use it for logging the client's console output.

1. Open a PuTTY session to the system that manages the LPAR and log in with a user ID that has the authority to open a console on the client. For example, log in to the HMC as hscroot.
2. Once the PuTTY session has been established, right-click the title bar of the PuTTY window and choose "Change Settings..." This will open the PuTTY Reconfiguration window for the session that you have opened.
3. On the left side, you will see a tree of Categories. Select the "Logging" category, which is listed under the "Session" category tree. You will see the "Options controlling session logging" options.
4. Under "Session logging:" you will choose the "Printable output" option.
5. Under "Log file name:" enter the file name that you want to save the console output to. This should start with your full PMR number, so that it can be easily associated with your PMR. For example: 12345.678.000.debug_boot.log. You will also want to click the "Browse..." option and choose a location to save the file, perhaps your Windows Desktop, for easy access.
6. Finally, click the "Apply" option to save your selections.

Logging the client console's output from an LPAR using the NIM master

If you don't or can't use PuTTY, or your telnet/SSH client does not do logging, you can log the client's console output by using the script command on the NIM master.

1. Run the script command as follows, providing the file name where you want log the client's console output:
   # script /tmp/12345.678.000.debug_boot.log
2. Once the script command is running, you should telnet or SSH to the system that manages the LPAR and log in as a user that has the authority to open the console on the client. Be sure to establish the connection from within the script session or the output will not be logged.
3. Proceed with the debug boot as described below.
4. Once you have reproduced the problem and have collected the debug boot, you will need to close the client's console window as described in the relevant section below, depending on which type of system manages the LPAR.
5. You will be returned to the command prompt on the system that manages the LPAR, where you should log out of that system.
6. You will now be at the AIX prompt on your NIM master, but the script session is still active. Exit the script session by typing:
   # exit

 

---

Opening the client's console

Once you have enabled logging in the debug boot session, you will need to open the client system's console in this session. How this is accomplished depends on what type of system manages the client LPAR. If the client is not an LPAR, but is a stand-alone system, no action is required, as the console output will be sent to the S1 serial port on the system.

Opening the client's console on an HMC managed LPAR

If your LPAR is managed by a Hardware Management Console (HMC):

1. Telnet or SSH to the HMC and log in as hscroot.
2. Once logged in, open the console on the client LPAR using either the vtmenu or mkvterm commands:
   1. :~> mkvterm -m managed_system -p partition_name
   2. :~> vtmenu
      • Choose the number of the Managed System
      • Choose the number of the client LPAR

Opening the client's console on an IVM managed LPAR

If your LPAR is managed by an Integrated Virtualization Manager (IVM):

1. Telnet or SSH to the IVM and log in as padmin.
2. Once logged in, open the console on the client LPAR using the mkvt command:
   • mkvt -id partition_number

 

---

Performing the Debug Boot

Now that you have enabled logging of the console, you need to perform the debug boot.

1. On the NIM master, set up the operation that you are having a problem with. This may be a bos_inst or maint_boot operation. Use the same NIM resources that you used when the problem you are debugging was encountered. This can be done via AIX command line or using smitty.
2. Boot the client in System Management Services (SMS) mode.
   1. For stand-alone systems, this is done by powering the system on and waiting for the following to be displayed:
      IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
      
         1 = SMS Menu                   5 = Default Boot List
         8 = Open Firmware Prompt       6 = Stored Boot List
      
         Memory      Keyboard     Network     SCSI     Speaker
      
      Note that there will be a slight pause of one or two seconds after the word "Keyboard" is displayed and before "Speaker" is displayed. You will also hear a set of tones if you are near the system. This is the point where you need to press the 1 key to enter SMS. If you hear another set of tones, or see the word "Keyboard" appear before pressing the 1 key, then you have missed the window to enter SMS and you will have to power the system off and on to try again.
      
      Note: On some systems with a graphics console, you may need to press the F1 key instead of the 1 key. Press the key that is displayed indicating SMS.
   2. For IVM managed systems, you will activate the LPAR from the IVM instead of powering the system on. Otherwise, the procedure is the same, press the 1 key on the client's console after the word "Keyboard" is displayed, but before "Speaker" displays.
   3. For HMC managed systems, you may also use the above procedure, but there is also a way to activate the LPAR in SMS mode:
      1. If the LPAR is not in the "Not Activated" mode, perform a shutdown operation on the LPAR.
      2. Select the client LPAR by checking the box next to it in the HMC GUI.
      3. Select the Operations --> Activate --> Profile option from the Tasks menu.
      4. In the "Activate Logical Partition" dialog, choose the correct "Logical Partition profile", then click the "Advanced..." option. Do not check the "Open a terminal window or console session" box, since we already have a console open.
      5. In the "Activate Logical Partition - Advanced" dialog, choose "SMS" from the "Boot mode" menu. Click OK.
      6. Finally, click OK to activate the LPAR. It will automatically enter SMS.
   4. If you were successful, you will see a menu similar to this in the window which is now being logged:
       Version AL730_149
      SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.
      ----------------------------------------------------------
      Main Menu
      1.   Select Language
      2.   Setup Remote IPL (Initial Program Load)
      3.   Change SCSI Settings
      4.   Select Console
      5.   Select Boot Options
      
      ----------------------------------------------------------
      Navigation Keys:
              X = eXit System Management Services
      ----------------------------------------------------------
       Type menu item number and press Enter or select Navigation key:
3. Next, you will verify that the network information that SMS will use to perform the network boot is correct.
   1. From the SMS Main Menu, choose the "Setup Remote IPL (Initial Program Load)" option.
   2. Next, select the network adapter that will be used to perform the network boot. If you're not sure which adapter to use, the Location Code or MAC address may help you identify it. Otherwise, you will have to guess and see if it works by performing a ping test in a later step.
   3. Select whether you are using IPv4 or IPv6.
   4. Select "BOOTP"
   5. Select "IP Parameters"
   6. Verify that the information on this screen is correct. If not, you may make corrections by selecting each item. Note that the "Gateway IP Address" and "Subnet Mask" should be the client's gateway and subnet mask.
   7. Once these values are correct, press ESC to return to the previous menu.
   8. Select the "Ping Test" option.
   9. Again, verify that the information on this screen is correct for your network configuration. There are additional values on this screen that may be changed on the "Adapter Configuration" menu from the previous menu, but most of the time this information does not need to be changed.
   10. If the information on the Ping Test screen is all correct, select the "Execute Ping Test" option. The ping test should be successful.
   11. If the ping test was not successful, you will have to troubleshoot the problem. Verify that the IP information and adapter configuration are correct. Also, make sure that you have chosen the correct network adapter, if you have more than one. Contact your Support Technician, if necessary.
       If the ping test was successful, press "m" to return to the SMS Main Menu.
   12. From the SMS Main Menu, choose the "Select Boot Options" option.
   13. Select "Configure Boot Device Order"
   14. Select "Select 1st Boot Device"
   15. Select "Network"
   16. Select "BOOTP"
   17. Enter the number of the network adapter that was configured and has a successful ping test.
   18. Select "Set Boot Sequence: Configure as 1st Boot Device"
   19. The Current Boot Sequence will be displayed. Verify that the correct network adapter is displayed as the first boot device.
   20. If the Current Boot Sequence is not correct, return the the previous menu and correct the problem. Otherwise, press "m" to return to the SMS Main Menu.
   21. At the SMS Main Menu, enter the hidden menu option, 0 and press Enter and confirm that you want to exit SMS. This will drop you into Open Firmware.
   22. At the 0 > prompt, enter boot -s verbose and press enter. For the newest power9 firmware with the restricted open firmware prompt, use the new command “BOOT_FROM_SEQ -s verbose”
   23. The debug boot will begin and be displayed to the console.

 

---

Closing the client's console and submitting the log file

Now that you have collected the requested debug output, you just need to close the client's console and submit the log file that you created. To close the client's console, use the appropriate method described below. To submit the log file, please follow the instructions provided by your Technical Support Technician.

Closing the client's console on an HMC or IVM managed LPAR

If your LPAR is managed by either an HMC or IVM system, you can close the console using the following series of three keystrokes:
[Enter][tilde(~)][period(.)]

This will return you to the HMC or IVM command prompt. You may then log out of the HMC or IVM system, and terminate logging.

Submitting the log file to Technical Support

You may now submit the collected to to the Technical Support team using the instructions that you were provided. If you were not provided instructions for submitting the data, try using one of the methods described here:
https://www.ibm.com/support/docview.wss?uid=isg3T1022758