Netbooting on POWER - An Introduction

2 likes Updated 6/30/16, 3:47 AM by OneSkyWalkerTags:

Introduction


In conversations with perspective Linux on POWER clients, we are often asked if POWER supports PXE boot. The short answer is “not exactly”, because a true PXE environment has some x86 architecture specific functions/extensions within it. That said, we do support the same basic functions and underlying protocols as PXE. And now that POWER , beginning with Fedora 17, has moved to grub2 for a bootloader, the appearance and function are quite similar. In other words, it is functionally equivalent to PXE. For example, like PXE, grub2 supports tiered menus and the ability to edit those menus from the client side.

In the following article, we will discuss the current state of netbooting Linux on POWER and then demonstrate a simplified setup using Fedora 18.

Components for netbooting


For most people, PXE is really a number of network servers and protocols that allow for a client to get onto a network and obtain content like installation images and files. One of the main purposes is for dynamic or automatic installation of the client without the use of media such as thumb drives or optical media.

The basic infrastructure components needed for netbooting are a DHCP server, a TFTP server, and some form of a file server. The following figure represents a logical flow of how the client interacts with these network servers when netbooting with grub2.



When a typical network client powers-up and seeks to boot from the network, it will initially send a DHCP request to the network. The DHCP server provides the client with the proper IP information (IP address, subnet mask, gateway address, domain server address) and minimally a boot file location/pathname. For POWER, the DHCP server must be configured to allow BOOTP clients as OpenFirmware on POWER systems still requires BOOTP and cannot do a pure DHCP implementation.

When netbooting with grub2, the bootfile is actually the grub2 binary itself. The client will attempt to get the binary using TFTP. When the TFTP serves this binary, the client will then execute it and grub2 will begin to load. During this execution, grub2 will attempt to obtain its configuration file. This configuration file contains references for:

  • grub2 modules
  • the location for the kernel and initrd
  • the protocol that should be used to obtain the kernel and initrd
  • additional kernel arguments


Additionally, any modules in the grub2 configuration will be loaded at this time. Grub2 will continue to use TFTP request and load these modules.

Then grub2 will attempt to fetch the kernel and initrd (in this example). Grub2 currently supports the http and tftp protocols. In the above illustration, this content is obtained with the http requests.

 

An Example: Installing Fedora 18 using netboot

We can demonstrate the actual setup and configuration for netbooting with an example. In this example we choose to use Fedora 18 as our Linux distribution because it incorporates a feature rich grub2 for POWER, is a known development distribution for our work in grub2, contains all the necessary network servers and services, and is freely available. But most importantly, Fedora 18 contains the most recent grub2 package with POWER specific contributions. We will use an existing Fedora 18 server to netboot and install Fedora 18 to a new node.

There is undoubtedly ample opportunity to create a more advanced scenario,which may be ultimately more reflective of reality, but we have reserved that for subsequent articles and write-ups. We have knowingly chosen a simple scenario for our example because it allows us to focus on something that is easily reproducible by others. The following illustration shows a simple, single-subnet network with a single network client and a single network services system.





The single network system provides the DHCP (with BOOTP), HTTP, and TFTP services. We have chosen HTTP again for our file serving. The NIC in the install node has a MAC address of 7a:ed:b0:cb:03:04 which will be relevant for setting up the DHCP/BOOTP components.


Install the required software

 

You can use yum to install the additional software you will need on the network server. In our case, you will want to use yum to install dhcp, tftp-server, and httpd. Yum will automatically resolve and install any package dependancies. Optionally, you may wish to install tftp.


[root ~] # yum install tftp-server dhcp http tftp


Setting up the TFTP server


In Fedora, the tftp server is controlled by the xinetd daemon (xinet.d). Furthermore, when tftp-server is installed, it defaults to off (or disabled). To enable it, you must change the tftp-server configuration file (/etc/xinetd.d/tftp) and change the disable parameter. The following is a functional configuration for the tftp-server.

service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /var/lib/tftpboot -v
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}


Firewalld, Fedora's firewall application, blocks inbound tftp requests by default. To allow firewalld to run and use the tftp-server, you can issue the following command:

firewall-cmd --permanent --add-service=tftp

You now must start (or restart) the tftp-server.

systemctl start|restart xinetd

Setting up the HTTP Server


The basic HTTP server setup in Fedora 18 is quite simple. In this scenario and configuration, the http server is used to provide the Fedora 18 kernel and initrd as well as the installer and its required files. The http root in the filesystem is /var/www/html. The easiest way to serve this content is with http is to mount the Fedora iso image in the http root on the filesystem itself. For example:

[root ~] # mkdir /var/www/html/f18/
[root ~] # mount -o loop /root/Fedora-18-Beta-ppc64-DVD.iso /var/www/html/f18/


Now the http server needs to be started.

systemctl start httpd
 

Configuring grub2 on the network server


By default, Fedora 18's tftp server will serve content from /var/lib/tftpboot/ as defined by the server_args -s switch in /etc/xinetd.d/tftp. We will now refer to /var/lib/tftpboot as the root tftp path. You now need to put grub2 and its supporting files in the root tftp path so that the client nodes can obtain them. Grub2 provides a utility to “package” the necessary files for netbooting and minimally only needs a path to the root tftp location.

[root ~] # grub2-mknetdir –net-directory=/var/lib/tftpboot
Netboot directory for powerpc-ieee1275 created. Configure your DHCP server to point to /boot/grub2/powerpc-ieee1275/core.elf


As the message indicates, /boot/grub2/powerpc-ieee1275/core.elf should be defined in your DHCP configuration with the filename parameter. It is actually the grub2 binary itself.

[root ~] # file /var/lib/tftpboot/boot/grub2/powerpc-ieee1275/core.elf
/var/lib/tftpboot/boot/grub2/powerpc-ieee1275/core.elf: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), statically linked, stripped


Now that grub2 and its required files are in their proper locations, we need to create the configuration file for grub2. As mentioned before, the grub2 configuration file contains the parameters, among other things, that tell grub2 where to get it's kernel and initrd. For this example, we will also add the required kernel arguments for the Fedora installer itself. The grub2 configuration file needs to be within the tftp root path and specifically at /var/lib/tftpboot/boot/grub2/grub.cfg. The following is a configuration we used to install Fedora 18.
 

menuentry 'Install Fedora 18' {
insmod http
insmod tftp
set root=http,192.168.1.1
echo 'Loading F18 Install kernel ...'
linux /f18/ppc/ppc64/vmlinuz repo=http://192.168.1.1/f18 inst.vnc
echo 'Loading initial ramdisk ...'
initrd /f18/ppc/ppc64/initrd.img
}

You can add multiple menu entries in a single configuration as well. This is handy for being able to install multiple distributions or a single distribution differently. Also note the use of “set root=http, 192.168.1.1”. This instructs grub2 to use HTTP and the 192.168.1.1 as its root path for obtaining content thereafter. All the rest of the paths are then relative HTTP paths. Kernel arguments are passed after the 'linux' parameters. Fedora 18 network installs require the use of a “repo” argument. The “inst.vnc” argument enables the Fedora installer to be run over vnc.


Setting up the DHCP server

You must now edit the configuration file for dhcpd. For this simple scenario, a configuration like below would be sufficient:

subnet 192.168.1.0 netmask 255.255.255.0 {
allow bootp;
option routers 192.168.1.3;
group { #BOOTP POWER clients
filename "boot/grub2/powerpc-ieee1275/core.elf";
host client1 {
hardware ethernet 7a:ed:b0:cb:03:04;
fixed-address 192.168.1.101;
}
}
}

Note how each host must have a hardware ethernet (MAC) address and a fixed IP address. We were not able to perform dynamic addressing in this scenario. We have used the group parameter to organize POWER clients which require BOOTP support. Because the filename attribute is associated with the group, the filename is inherited by each host definition. You can override this behavior by adding a filename attribute to a host.

You can now start or restart your DHCP server.

systemctl start|restart dhcpd


Booting the client


By default, POWER systems are set to boot from the network by default. This behavior can be modified in SMS, which is akin to x86 architecture BIOS menus. The behaviors and use of SMS is outside the scope of this article but is being considered as an advanced topic or follow-on content.

Please note that SMS requires user interaction. The HMC lpar_netboot command provides a similar capability without requiring user interaction beyond issuing the command.

The following shows the early netboot messages for a POWER system and ends with the kernel beginning to execute.


IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

1 = SMS Menu 5 = Default Boot List
8 = Open Firmware Prompt 6 = Stored Boot List


Memory Keyboard Network SCSI Speaker
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM STARTING SOFTWARE IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM PLEASE WAIT... IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM







TFTP BOOT ---------------------------------------------------
Server IP.....................192.168.1.1
Client IP.....................192.168.1.101
Subnet Mask...................255.255.255.0
( 1 ) Filename.................boot/grub2/powerpc-ieee1275/core.elf
TFTP Retries..................5
Block Size....................512
FINAL PACKET COUNT = 2880


GNU GRUB version 2.00

+--------------------------------------------------------------------------+
|Install Fedora 18 |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------------+

Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.


Loading F18 Install kernel ...
Loading initial ramdisk ...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.6.6-3.fc18.ppc64 (mockbuild@) (gcc version 4.7.2 20121001 (Red Hat 4.7.2-3) (GCC) ) #1 SMP Fri Nov 9 11:20:36 MST 2012
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 256 (NR_CPUS = 1024)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/f18/ppc/ppc64/vmlinuz repo=http://192.168.1.1/f18 inst.vnc
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 00000000069a0000
alloc_top : 0000000010000000
alloc_top_hi : 0000000010000000
rmo_top : 0000000010000000
ram_top : 0000000010000000
instantiating rtas at 0x000000000ee90000... done
Querying for OPAL presence... not there.
boot cpu hw idx 0
copying OF device tree...
Building dt strings...
Building dt structure...


From here, the kernel and initrd load. Eventually it will load the Fedora installer and you will have a very similar experience as any other architecture.


Useful debugging techniques

 
tcpdump
You can use the network utility tcpdump to monitor ports on the host machine. This is very handy when trying to debug the netbooting process given the numerous network services. Here is an example of some output showing our single, client node netbooting.

[root ~] # tcpdump -i eth0 port 68 or port 69 or port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:22:08.515263 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 7a:ed:b0:cb:03:04 (oui Unknown), length 300
10:22:08.515492 IP .bootps > 192.168.1.101.bootpc: BOOTP/DHCP, Reply, length 300
10:22:18.666503 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 7a:ed:b0:cb:03:04 (oui Unknown), length 300
10:22:18.666651 IP .bootps > 192.168.1.101.bootpc: BOOTP/DHCP, Reply, length 300
10:22:28.802906 IP 192.168.1.101.48198 > .tftp: 45 RRQ "boot/grub2/powerpc-ieee1275/core.elf" octet
10:22:36.792317 IP 192.168.1.101.25300 > .tftp: 70 RRQ "//boot/grub2/powerpc-ieee1275/normal.mod" octet blksize 1024 tsize 0
10:22:36.805736 IP 192.168.1.101.25301 > .tftp: 72 RRQ "//boot/grub2/powerpc-ieee1275/terminal.mod" octet blksize 1024 tsize 0
10:22:37.007334 IP 192.168.1.101.25302 > .tftp: 70 RRQ "//boot/grub2/powerpc-ieee1275/crypto.mod" octet blksize 1024 tsize 0
10:22:37.208346 IP 192.168.1.101.25303 > .tftp: 71 RRQ "//boot/grub2/powerpc-ieee1275/gettext.mod" octet blksize 1024 tsize 0
10:22:37.409633 IP 192.168.1.101.25304 > .tftp: 68 RRQ "//boot/grub2/powerpc-ieee1275/gzio.mod" octet blksize 1024 tsize 0
10:22:37.411326 IP 192.168.1.101.25305 > .tftp: 40 RRQ "//grub.cfg" octet blksize 1024 tsize 0
10:22:37.785333 IP 192.168.1.101.25306 > .tftp: 71 RRQ "//boot/grub2/powerpc-ieee1275/command.lst" octet blksize 1024 tsize 0
10:22:37.986560 IP 192.168.1.101.25307 > .tftp: 66 RRQ "//boot/grub2/powerpc-ieee1275/fs.lst" octet blksize 1024 tsize 0
10:22:38.187362 IP 192.168.1.101.25308 > .tftp: 66 RRQ "//boot/grub2/powerpc-ieee1275/fs.lst" octet blksize 1024 tsize 0
10:22:38.388366 IP 192.168.1.101.25309 > .tftp: 70 RRQ "//boot/grub2/powerpc-ieee1275/crypto.lst" octet blksize 1024 tsize 0
10:22:38.785350 IP 192.168.1.101.25310 > .tftp: 70 RRQ "//boot/grub2/powerpc-ieee1275/crypto.lst" octet blksize 1024 tsize 0
10:22:38.986403 IP 192.168.1.101.25311 > .tftp: 72 RRQ "//boot/grub2/powerpc-ieee1275/terminal.lst" octet blksize 1024 tsize 0
10:22:39.187351 IP 192.168.1.101.25312 > .tftp: 72 RRQ "//boot/grub2/powerpc-ieee1275/terminal.lst" octet blksize 1024 tsize 0
10:22:39.388366 IP 192.168.1.101.25313 > .tftp: 51 RRQ "//boot/grub2/grub.cfg" octet blksize 1024 tsize 0
10:22:45.896478 IP 192.168.1.101.25314 > .tftp: 68 RRQ "//boot/grub2/powerpc-ieee1275/echo.mod" octet blksize 1024 tsize 0
10:22:46.097546 IP 192.168.1.101.25315 > .tftp: 69 RRQ "//boot/grub2/powerpc-ieee1275/linux.mod" octet blksize 1024 tsize 0
10:22:46.299543 IP 192.168.1.101.25316 > .tftp: 67 RRQ "//boot/grub2/powerpc-ieee1275/elf.mod" octet blksize 1024 tsize 0
10:22:46.500602 IP 192.168.1.101.25317 > .tftp: 68 RRQ "//boot/grub2/powerpc-ieee1275/http.mod" octet blksize 1024 tsize 0
...


tail -f
It is helpful to monitor a couple of log files during the setup of the network services. In particular, we like to log /var/log/messages and /var/log/http/access_log with tail and the -f switch.

tftp-client
You can test the tftp-server on the host node by running a tftp client locally to fetch a file. If the host node cannot locally fetch a file, you likely have a service or path issue.
 

Conclusion and follow-on topics


With the migration to grub2 in Fedora, there is now a more feature rich netbooting capability for POWER systems. You can setup a netbooting environment by installing and configuring a few network services and because grub2 is very modular and flexible, your netbooting environments can become quite sophisticated.
This article was written to demonstrate a simple netbooting environment. We can envision more complex environments where the network services are physically or logically separated. Additionally, much of the netbooting and installs can be automated with functions like kickstart. And of course, interesting and useful boot environments could be derived from the ability and flexibility that netbooting provides. There topics and more are under consideration for future write-ups.

Paulo Flabiano Smorigo is a software engineer in LTC Brazil and currently works with GRUB for POWER machines. Paulo has worked with open source projects and embedded systems using Linux for almost a decade.

Brent Baude is a senior software engineer with IBM located in Rochester, MN. Brent currently works with Red Hat on IBM's behalf on the x86, s390, and POWER architectures. He is a long-standing contributor in the open source world and spends time helping open-source distributions like Gentoo and Fedora.