IBM PowerKVM on IBM POWER8 (Part 1)

System bring-up

This tutorial takes the reader through the details involved in bringing up an IBM® Power Systems™ server (that is based on IBM POWER8™ processor technology) with Open Power Abstraction Layer (OPAL) and IBM PowerKVM™ stack. This article also covers the upgradation of firmware and various host operating system (OS) installation methods.

Share:

Walesa Francis (walesafran@in.ibm.com), System Software Engineer, IBM

Photo of Walesa Francis Walesa is part of PowerKVM Functional Verification Team. He has the primary responsibility for the quality of PowerKVM and OPAL. He has a total experience of 10 years in Linux, virtualization, and Power Systems with the recent 8 years specifically on IBM Power and virtualization with India Systems and Technology Labs, Bangalore.



Manas K Nayak (maknayak@in.ibm.com), Senior Staff Software Engineer, IBM

Photo of Manas K NayakManas is currently a senior staff software engineer working as a functional verification test (FVT) specialist, where he focusses on end-to-end FVT of KVM on IBM Power Architecture. He has experience in testing IBM System x servers and Power Systems servers with enterprise distros such as Red Hat and SUSE, including community distros such as fedora, Red Hat clustering, and high availability (HA) solutions. He has supported many customer engagements related to installation issues and Red Hat SAN storage HA on X servers. He has a master degree in Computer Applications from Utkal University, Odisha. He has been part of India Systems and Technology Labs, Bangalore, India for the past 8 years.



Divya Vikas (divya.vikas@in.ibm.com), System Software Engineer, IBM

Photo of Divya VikasDivya is working as a LTC functional verification test specialist for components such as RAS, I/O adapters, and PowerKVM. In her earlier role, she has provided active support to customers across the world on AIX Server Message Block File System (SMBFS) and Fast Connect components. Currently her focus is on PowerKVM end-to-end FVT tests. She is part of India Systems and Technology Labs, Bangalore, for past 8 years.



19 June 2014

Also available in Chinese

Introduction to PowerKVM

Linux® has its own built-in hypervisor, kernel-based virtual machine (KVM). KVM support is now added to IBM Power Architecture, thereby allowing to virtualize IBM Power Systems using the open source virtualization standard. KVM being there for a while as an open source hypervisor for different architecture, now got ported and tuned to IBM Power Architecture®. Guests of leading Linux distribution vendors SUSE and Red Hat, can be deployed over enterprise-ready PowerKVM environment easily. IBM has also developed OPAL with engineering level of quality to run KVM on the next generation of IBM POWER® processor-based servers. Thus, IBM extend its smart hand to provide customer KVM as a dependable open source virtualization technology on Power Systems.

To understand PowerKVM, you need to understand the building blocks involved in Power Systems with PowerKVM. The following figure depicts the high-level blocks involved in Power Systems running PowerKVM.

Figure 1. Building blocks of PowerKVM
Building blocks of PowerKVM

Flexible service processor (FSP)

The FSP is an advanced service processor that provides function to initialize the system chip sets, boot to a hypervisor run time, interface to the management module, and provide runtime support for advanced reliability, availability, and serviceability (RAS) functions. FSP is an additional layer in the hardware to closely monitor the other hardware components, and during the event, it notifies the system administrator or vendor to work proactively before the failure to avoid disruption in the service and so on.

Firmware or OPAL

OPAL is the system firmware in the stack of POWER8 processor-based server, PowerKVM. It consists of three major parts called skiboot, skiroot, and skirun.

OPAL boot (skiboot)

This is a binary component (a Load ID) loaded by the FSP which performs some system initialization and setup (PCIe and so on) and provides runtime services abstraction to a client operating system. Those services range from accessing the realtime clock, rebooting the system, various PCIe PCI Host Bridging (PHB) related functions and nvram. OPAL additionally loads and transfers control to the OPAL loader during boot.

OPAL root (skiroot)

This a Linux-based boot loader which discover I/O devices and loads Linux-based hypervisor from disk or network

OPAL runtime (skirun)

OPAL runtime is a linux system (kernel + ramdisk) stored in a separate local ID. This system is used to access I/O devices (such as disks, network, and so on) and locate and load the host Linux system to be run on the system in hypervisor mode.

PowerKVM

PowerKVM is the Linux host kernel that runs in the hypervisor mode . PowerKVM, with the help of an open source KVM ecosystem, allows to run multiple Linux guests seamlessly in the supervisor mode. Host kernel has access to all memory and all system resources. All the guests running on top of PowerKVM are paravirtualized using the PAPR interface. The PowerKVM host runs in the hypervisor mode where the guest kernel runs in the supervisor mode. The special firmware, OPAL, is developed to provide full access to the hypervisor mode and guests can run in full hardware speed without emulating any privileged instructions.


Interfaces to FSP

The users can access the FSP of Power Systems through IPMItool and Advanced System Management (ASM).

The IBM recommended and advertised way to access FSP is through the open source command line based IPMItool. Users can also use the GUI provided by FSP known as ASM.

IPMItool:

IPMItool is an open source system administration tool which can be downloaded and installed on a Linux system. After the installation, the FSP can be accessed using the following command.

 $ipmitool -I lanplus -H <fsp_ip> -P <fsp_password> power status

Where,

  • fsp_ip: Denotes the IP of FSP
  • fsp_password: IPMI password of FSP.
    A default password would be set to 'passw0rd'. Refer to the firmware release notes for more information.
  • power: Is an example of command IPMITool supports on Power Systems.

Check the IPMItool manual page for other commands and usage.

ASM:

ASM is a GUI-based web offering from FSP for managing systems. Users can connect to the ASM of the respective FSPs using https://<fsp_ip> on any browser.


Selecting the hypervisor

Before 2014, POWER firmware was a package of firmware (software), which ran on FSP and IBM POWER Hypervisor™. Now POWER firmware would also include the load IDs of OPAL (firmware for PowerKVM) along with FSP firmware and POWER Hypervisor.

This would mean that the firmware required for PowerKVM and IBM PowerVM stack would always be available in a system post 2014. Therefore, when the Power Systems servers are being switched on, it is imperative that users need to select the firmware based on their stack requirement. This step can be accomplished with ASM.

Perform the following steps to select a hypervisor using ASM:

On the left side panel, click System Configuration ➟ Hypervisor Configuration. Then select OPAL.

Note:

  • This option would be available only when the system is in the Powered Off state.
  • Setting of hypervisor to OPAL has to be done when there are no HMCs attached. So, detach the HMC connection, if any, before setting the hypervisor to OPAL.

PowerKVM system bring-up and installation steps

  1. Update firmware on the Power System server
  2. Configure PowerKVM repository
  3. Switch on the POWER8 H/W system
  4. Select the image to boot from Petitboot
  5. Install PowerKVM
  6. Log in to PowerKVM
  7. Access the installed system

1. Update firmware on the Power Systems server

There may be instances when the user might have to upgrade the Power Systems firmware to a more recent level to acquire new features or additional support for devices. PowerKVM provides a facility to do that. Make sure that the following requirements are met.

  • An OS to be running on the system (so, proceed with other topics to start the OS before coming back for firmware updation).
  • The .img file of the OPAL level that the user needs to update to.

Power Systems has two sides of flash to boot firmware from, namely permanent and temporary. This provides a way to test firmware updates on the temporary side of the flash before committing the tested changes to the permanent side, thereby committing the new updates.

Perform the following steps for the update:

  1. Save the level of the existing firmware before really updating. In ASM, in the system menu, click Service Aids ➟ Service Processor Command Line, and run the following command:
     cupdcmd -f
  2. Download the .img file of the level of firmware to be updated to a location in PowerKVM. Refer to IBM Fix Central for downloading the image file.
  3. Verify the image downloaded by running the following command and save the output.
     $update_flash -v -f <file name.img>
  4. Update the firmware by running the following command.
     $update_flash -f <file name.img>

    Note:

    • The command reboots the system and therefore, sessions if any, would be lost.
    • Do not reboot or switch off the system until it is back.
  5. Verify the updated firmware level of the temporary side of the flash as in step 1.
  6. The new updated level can be committed to the permanent side of the flash by running the following command.
    $update_flash -c

In case the update has to be reverted, the user can do so by running the $update_flash -r command (rejection would reject only the temporary side of the flash. Therefore, the new level should be committed to the permanent side only after thorough testing of the new firmware).

2. Configure PowerKVM repository

A. Download PowerKVM build

In order to download the PowerKVM product, refer to the IBM Fix Central web page

B. Configure repository

You can configure different types of repositories for installation.

Netboot/PXEboot repository setup

For this setup, you will need a Dynamic Host Configuration Protocol (DHCP) / Trivial File Transfer Protocol (TFTP) server and a web server.

  1. Download the ibm-powerkvm-xxx-ppc64-GA-xxx.iso file and do a loop mount it to unpack it into some folder within your www root structure of your HTTP server .
    $  mkdir /tempiso
    Download  ibm-powerkvm-xxx-ppc64-GA-xxx.iso to /tempiso
    $ cd /tempiso
    $ mkdir iso
    $ mount -o loop ibm-powerkvm-xxx-ppc64-GA-xxx.iso ./iso
  2. Copy the content of the upacked directory to your web server document root directory.
    $ mkdir /www-root-directory/powerkvmnetboot
     For example: mkdir /var/www/html/powerkvmnetboot
    $ cp -a ./iso/* <www-root-directory/powerkvmnetboot>
     For example: $ cp -a ./iso/* /var/www/html/powerkvmnetboot
  3. Create the petitboot.conf file to your tftproot directory say /tftproot/kvmonp
    $ mkdir -p /tftproot/kvmonp 
    $ touch /tftproot/kvmonp/petitboot.conf
  4. Adjust the petitboot.conf file to your web server configuration path and IP address.
    $ vi /tftproot/kvmonp/petitboot.conf    
    label PowerKVMAutomatedInstall_BUILD13
    kernel http://YOUR-SERVER-IP/powerkvmnetboot/ppc/ppc64/vmlinuz
    initrd http://YOUR-SERVER-IP/powerkvmnetboot/ppc/ppc64/initrd.img
    append root=live:http://YOUR-SERVER-IP/powerkvmnetboot/LiveOS/squashfs.img 
    repo=http://YOUR-SERVER-IP/powerkvmnetboot/packages rd.dm=0 rd.md=0 
         console=hvc0 console=tty0

    Save the peitboot.conf file with above content.
  5. Edit your dhcpd.conf file to set to point to the peitboot.conf file contents and configurations for PXE boot.

    Set the following directive at the beginning

    allow bootp;
    allow booting;
    ddns-update-style none;
    always-reply-rfc1048 true;
    ignore unknown-clients;
    option conf-file code 209 = text;
  6. Append the system directive to the /etc/dhcpd/dhcpd.conf file.
    host <your-system> { 
                 hardware ethernet <system macaddr>
                 fixed-address <system ip>; 
                 option host-name "<system hostname>";
                 option conf-file "<kvmonp/petitboot.conf>"; }

    For example:
    host powerkvm.host.domain.com { 
               hardware ethernet 52:53:88:60:12:40; 
               fixed-address 192.168.10.149; 
               option host-name "powerkvm.host.domain.com"; 
               option conf-file "<kvmonp/petitboot.conf>"; }
  7. Save the dhcpd.conf file.
  8. Restart the dhcpd service or reboot the DHCP server.
    $service dhcpd restart

Network File System (NFS) repository setup

  1. Configure your TFTP and DHCP server and copy the kernel (vmlinuz) and initrd into it as covered in Netboot.
  2. Configure your NFS server.
  3. Copy the PowerKVM ISO into a NFS folder.
  4. Adjust your petitboot.conf file to your web server configuration.
    label PowerKVMAutomatedInstall_BUILD13
    kernel http://YOUR-SERVER-IP/powerkvmnetboot/ppc/ppc64/vmlinuz
    initrd http://YOUR-SERVER-IP/powerkvmnetboot/ppc/ppc64/initrd.img
    append root=live:http://YOUR-SERVER-IP/powerkvmnetboot/LiveOS/squashfs.img
    repo=http://YOUR-SERVER-IP/powerkvmnetboot/packages rd.dm=0 rd.md=0 
         console=hvc0 console=tty0

CD/DVD boot setup

  1. Use the downloaded PowerKVM ISO image to burn into a blank CD/DVD.
  2. Begin installation when you see the "POWERKVM_LIVECD" message on the petitboot screen.

HTTP boot setup

You need have a web server running where the content of the netboot.tar file or ISO is available and accessible over the network. This content could be available for Hypertext Transfer Protocol (HTTP) boot at: http://<your httpserver.com>/powerkvmnetboot

To use the HTTP boot method, we need to perform the following steps.

  1. Clear out any BSO access to your HTTP repository server.
  2. In the petitboot menu, press n to create a new temporary entry.
    Fill out the entry as mentioned below:
    Kernel: http://YOUR-http-SERVER-IP/powerkvmnetboot/ppc/ppc64/vmlinuz
    Initrd: http://YOUR-http-SERVER-IP/powerkvmnetboot/ppc/ppc64/initrd.img
    Device tree: ---------------------------------------------------------
    Boot arguments: 
      root=live:http://YOUR-http-SERVER-IP/powerkvmnetboot/LiveOS/squashfs.img
    repo=http://YOUR-http-SERVER-IP/powerkvmnetboot/packages rd.dm=0 rd.md=0 
      console=hvc0 console=tty0

3. Switch on the POWER8 H/W system

After the hypervisor has been selected and the firmware upgraded, if required, the POWER8 system can be powered on.

Perform the following steps to switch on.

Using ASM:

  1. In ASM click Power/Restart Control ➟ Power On/Off System.
  2. On the page, make sure that the boot options are selected appropriately.
    • Firmware boot side for the next boot: Temporary or Permanent
    • System operating mode: Normal
    • Server firmware start policy: Running
    • System power off policy: <As required>

System console

In the OPAL hypervisor mode, there are two methods for users to access the system console, which can be used for troubleshooting and tracking the bootup of systems.

Using the IPMI console:

The IPMItool provides Serial Over LAN (SOL) console of the system which can be accessed by the following command.

 $ipmitool -I lanplus -H <fsp_ip> -P <ipmi_password> sol activate

Refer to man page of the IPMItool for other options available with the sol command.

4. Select the image to boot from Petitboot

Petitboot is the bootloader for OPAL and PowerKVM stack on IBM Power Systems and part of "OPAL" in Figure 1.

It is a text-based tool that can scan the systems for all bootable instances on disks, network, and media. It lists the bootable instances for users to select the source to boot from. It also provides the option for users to create new temporary boot instances manually, which are typically used for HTTP boot.

It also provides a way to view and set the basic network configuration. The network configuration set here would reflect in the system after the installation of PowerKVM, if not overrun by network configuration during PowerKVM installation.

For example, the petitboot menu looks as shown in Figure 2.

Figure 2: Petitboot
Figure 2: Petitboot

5. Install PowerKVM

A. New installation

New installation would mean all the configuration files be erased and the selected disk be reformatted.

This step expects you to have followed all required repository setup before beginning with the installation.

Note that you can choose any of the methods to install as mentioned in repositories configuration part of this tutorial to install PowerKVM. Here, we specifically have screen captures for PXE installation involved at the time of writing.

Restart your Power Systems server and verify whether the "IBMPowerKVMAutomatedInstall" option is available at petitboot to begin installation.

Select the label to proceed further for installation.

Figure 3 : New installation
Figure 3 : New installation

On selecting IBMPowerKVMAutomatedInstall as shown in Figure 3, you can see boot messages and it takes a few minutes to check all existing Ethernet interfaces to detect. So, be patient and wait for the installation screen to be opened on the console.

In the language selection screen , retain the default selection to select English and press Enter to go to the next screen.

Figure 4 : Installation screen
Figure 4 : Installation screen

The screen to install IBM PowerKVM is displayed.

If the system is already having the PowerKVM OS installed , then you can see two options: Install IBM PowerKVM and Reinstall IBM PowerKVM. Otherwise, you will see only the Install IBM PowerKVM option. Select the option and press Enter to begin a fresh installation.

Then, follow the on-screen instructions as shown in the following screen captures to complete installation.

Figure 5: Disk selection

Figure 6: Root password
Figure 6: Root password
Figure 7. Interface selection
Interface selection
Figure 8: Interface configuration
Figure 8: Interface configuration
Figure 9: Installation
Figure 9: Installation

After reaching 100%, press Enter for the Reboot option to boot into the PowerKVM host system.

Then finally, follow the post-installation steps as mentioned in the later part of this tutorial.

B. Reinstallation

There is a possibility to reinstall the system, instead of a fresh install. When you select the reinstall option, the system will be erased by cleaning all the partitions and configurations and a full installation takes place. However, this method retains the contents in /var/lib/libvirt/images and /var/log .

To proceed with the reinstallation option, you need to perform the following steps:

  1. Boot the PowerKVM CD/DVD, or pxeboot, httpboot, or nfs method boot.
  2. Select the language as English and press Enter.
  3. Select Reinstall PowerKVM. This displays a warning message as shown in Figure 10.
    Figure 10: Reinstallation
    Figure 10: Reinstallation
  4. Click Yes to allow reinstallation to proceed.
  5. Wait for the process to finish installation and reboot when prompted.
  6. Press Enter on reboot to reboot the PowerKVM host .
  7. Then, proceed with the post-installation steps.

Note: Refer to IBM PowerKVM 2.1 Known Issues and Readme for limitations and known issues with reinstallation, if any, under PowerKVM installation considerations.

6. Log in to PowerKVM

After the installation is complete, the system will reboot. From the Petitboot menu, select one of the installed disk labels as given below:

  • IBM PowerKVM
  • IBM PowerKVM, with Linux 3.10.23-1300.pkvm2_1.1.ppc64

It boots from the installed kernel and sets the selinux policy. Later, it reboots automatically to save the configured selinux policy.

Now again, select one of the labels as mentioned above and then wait to see the login prompt on the IPMI console.

IBM_PowerKVM release 2.1.0 build 13 Beta5 (pkvm2_1) Kernel
3.10.23-1300.pkvm2_1.1.ppc64 on a ppc64 (hvc0) 
powerkvmhost login:

7. Access the installed system

You should be able to access the installed system from the console as well as remotely using Secure Shell (SSH) to the system IP.


Verifying the PowerKVM installation

To verify whether the installation was successful, you can use the following command:

$uname -a
Linux powerkvmhost.xxx.ibm.com 3.10.23-1400.pkvm2_1.2.ppc64 #1
SMP Thu Apr 17 15:32:52 CDT 2014 ppc64 ppc64 ppc64 GNU/Linux

The highlighted part should tell the level of PowerKVM installed.


Troubleshooting

This section covers the various errors and the tips for troubleshooting these errors.

  1. IPMItool commands returning:
     Error: Unable to establish IPMI v2 / RMCP+
    session Error: No response activating SOL payload

    Verify whether the hypervisor selection still holds good. If the FSP has been rebooted inadvertently, the hypervisor may go 00. It has to be 03 for IPMI commands to work.
  2. IPMItool command returning:
     Error: Unable to establish IPMI v2 / RMCP+
    session

    Verify and correct the IPMItool password through ASM.
  3. Not seeing anything in the IPMI console or debug TTY. Make sure that the system is powered on. The system stays in the Powered Off state after FSP reboot.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=974079
ArticleTitle=IBM PowerKVM on IBM POWER8 (Part 1)
publish-date=06192014