How To
Summary
The serial console is sometimes the unique method left you have handy to gain access to a shell attached to a terminal emulation program ready to be used to troubleshoot a system having issues.
It has proved to be an effective tool to assist you in tracking and spotting boot time events preventing the OS to boot and to gain access to the installed OS when the usual connectivity media via standard network interfaces are buggy or the network subsystem is broken. This is specially important in headless systems where not even a locally attached terminal is available.
Configuring a serial console has always been a tricky issue, and with the advent of the pervasive virtual data centers, doing it for isolated and standalone KVM guests injects an additional step of complexity.
We will cover in this article how to do it for a kvm guest running an instance of SLES 15 OS system and show you a way to use it for boot time troubleshooting purposes.
Objective
Hands on step by step procedure guide to configure a serial console device for a kvm guest running an instance of SLES 15 OS on x86_64 KVM Fedora based hypervisor host.
We will show at the very end how to use it to track boot time issues.
Environment
guest:
sles 15 SP1 kvm guest running the 4.12.14-197.7-default kernel.
host:
fedora 29 kvm host running the kernel 5.3.11-100.fc29.x86_64.
libvirt-4.7.0-3.fc29.x86_64
qemu-kvm-3.0.1-3.fc29.x86_64
libvirt-daemon-kvm-4.7.0-3.fc29.x86_64
Steps
Being able to access a SLES 15 instance via a serial console device requires your guest to meet the following requirements:
1) The virtual hardware for the guest supports and has been attached a serial line controller in the devices section of the xml hardware definition
2) The kernel running on the OS instance has built-in support for serial line controllers and the corresponding serial ports are correctly detected and identified at boot time
3) The boot loader installed in the OS instance is compatible and configured to tell the kernel about the supported serial consoles
In the following sections we will cover how to check and meet all these requirements so that you can connect to your guest using the serial console port.
Once those 3 steps are done, we will show you how to access the guest via the serial console and how to use it effectively to collect diagnostics data for a guest experiencing boot time issues.
1) Preparing the kvm guest hardware for a serial console
How do you know if a kvm guest has been attached a serial port controller and other hardware devices to support serial console access?
Run this command from the kvm host on the guest of your interest to find out:
[root@casld002 ~]# virsh -c qemu:///system dumpxml sles-15.1-efi-srv01 | grep -A 2 -E '\<console '
[root@casld002 ~]#
If you see an output like the one above then your guest does not support serial consoles.
You can do an additional check if the guest is still running and is accessible via ssh or a local console. If so, log in and run this command:
sles-151-efi-srv01:~ # dmesg | grep -E 'tty'
[ 0.004000] console [tty0] enabled
sles-151-efi-srv01:~ #
If you do not find any reference to a device featuring a name matching this "ttySn" pattern in the kernel ring buffer at boot time, the kernel is likely not detecting it.
If your guest is in a situation like the one depicted above then you have to edit its xml configuration and add the needed serial device controllers.
If you deployed the guest using default kvm client tools shipped in your KVM host like virt-install it is very unlikely that the resulting xml hardware definition lacks the sections corresponding to the serial devices. That said, you may have created the guest using other tools or simply the guest is one created long ago that you recently migrated to the current host and that needs some adjustments.
Since adding serial controllers to the device section of a kvm is not supported for a guest in a running status, you will have to stop it first if it happens to be up and running.
Note: In fact you can use the --config option of the virsh attach-device command, but the kernel won't detect the new devices until you reboot the guest.
Note: In fact you can use the --config option of the virsh attach-device command, but the kernel won't detect the new devices until you reboot the guest.
Once you have the guest cleanly shutdown and stopped, write a device section looking like this:
[root@casld002 ~]# cat /root/serial.xml
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
[root@casld002 ~]#
And now, run this command to attach the device to the sles-15.1-efi-srv01 guest:
[root@casld002 ~]# virsh -c qemu:///system attach-device sles-15.1-efi-srv01 --file /root/serial.xml --config
Device attached successfully
[root@casld002 ~]#
Now look for the "console" string again in the hardware definition for the guest:
[root@casld002 ~]# virsh -c qemu:///system dumpxml sles-15.1-efi-srv01 | grep -A 2 -E '\<console '
<console type='pty'>
<target type='serial' port='0'/>
</console>
[root@casld002 ~]#
If we take a closer look to the hardware definition of the guest we will see that two new devices were added:
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
That is exactly what we were after.
And that is all concerning our guest being able to support serial consoles.
2) Checking that kernel in the guest detects the serial console devices
Now we are ready to boot the guest and check again if the kernel manages to detect the serial devices and map the needed ports.
Now we are ready to boot the guest and check again if the kernel manages to detect the serial devices and map the needed ports.
Start the guest:
[root@casld002 ~]# virsh -c qemu:///system start sles-15.1-efi-srv01
Domain sles-15.1-efi-srv01 started
[root@casld002 ~]#
and gain access to a bash shell via ssh or a local console. Once logged in, run this command:
sles-151-efi-srv01:~ # dmesg | grep -E 'tty'
[ 0.004000] console [tty0] enabled
[ 3.562166] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
sles-151-efi-srv01:~ #
There you have the serial device and port correctly and successfully identified. More, you can check that there is a consistent and direct map between the number of the port as detected by the kernel and the port number used when defining the hardware for the guest in step 1.
If you look carefully the definition of the the hardware system while the guest is running you will see now that the guest has been assigned a serial port and a device file for a software terminal to attach to:
If you look carefully the definition of the the hardware system while the guest is running you will see now that the guest has been assigned a serial port and a device file for a software terminal to attach to:
[root@casld002 ~]# virsh -c qemu:///system list
Id Name State
----------------------------------------
1 ubuntu-16.4-dns01 running
2 fedora-gw running
3 centOS-7.2-iscsitarget running
4 ubunser-14.4-ldap1srv running
5 ubunser-14.4-nfssrv running
8 srv01-sles12sp3 running
11 sles-15.1-efi-srv01 running
[root@casld002 ~]#
[root@casld002 ~]# virsh -c qemu:///system dumpxml 11 | grep -A 11 -E '<serial'
<serial type='pty'>
<source path='/dev/pts/7'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/7'>
<source path='/dev/pts/7'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
[root@casld002 ~]#
If not sure, run this command as root from the guest:
sles-151-efi-srv01:~ # setserial -g /dev/ttyS[0123]
/dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
/dev/ttyS1, UART: unknown, Port: 0x02f8, IRQ: 3
/dev/ttyS2, UART: unknown, Port: 0x03e8, IRQ: 4
/dev/ttyS3, UART: unknown, Port: 0x02e8, IRQ: 3
sles-151-efi-srv01:~ #
That tells us the system only has one hardware port out of the 4 the kernel is built-in to detect by default.
In our system, the serial console must use the /dev/ttyS0 device, according to what was detected by the kernel above (see dmesg output).
sles-151-efi-srv01:~ # ls -l /dev/ttyS0
crw-rw---- 1 root dialout 4, 64 Jun 4 19:00 /dev/ttyS0
sles-151-efi-srv01:~ #
But that´s not enough to access the serial port. We need to tell the kernel to be prepared via the boot loader at boot time. Keep reading.
3) Telling the boot loader and kernel installed in the OS instance about the supported serial consoles
Now we need to tell the boot loader (grub in this case) about the serial consoles devices and interfaces we have configured in previous steps.
There are two main tasks to complete to have that configured properly.
Note: Consider getting a backup of the files before modifying them.
Note: Consider getting a backup of the files before modifying them.
a) Add/modify the following variables in /etc/default/grub configuration file:
From
GRUB_CMDLINE_LINUX_DEFAULT="splash=silent resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto quiet"
to
GRUB_CMDLINE_LINUX_DEFAULT="splash=silent resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto quiet console=tty0 console=ttyS0,115200n8"
From
GRUB_TERMINAL="gfxterm"
to
GRUB_TERMINAL="gfxterm console serial"
And finally add this new line after the GRUB_TERMINAL one:
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
If you want to have more verbose output, consider also removing "splash=silent" and "quiet" from the GRUB_CMDLINE_LINUX_DEFAULT variable.
b) Consolidate those settings defined in /etc/default/grub into the grub.cfg file:
For that you have to run the following command:
# grub2-mkconfig -o /boot/grub2/grub.cfg
You can now check the differences between the older (backed up) and the newer recently generated:
sles-151-efi-srv01:~ # diff /boot/grub2/grub.cfg.20200608 /boot/grub2/grub.cfg
76a77
> serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1
96c97,98
< terminal_output gfxterm
---
> terminal_input gfxterm console serial
> terminal_output gfxterm console serial
148c150
< linuxefi /boot/vmlinuz-4.12.14-197.7-default root=UUID=923d921d-0b7d-4cde-863f-c180d07ba58c ${extra_cmdline} splash=silent resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto quiet
---
> linuxefi /boot/vmlinuz-4.12.14-197.7-default root=UUID=923d921d-0b7d-4cde-863f-c180d07ba58c ${extra_cmdline} resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto console=tty0 console=ttyS0,115200n8
165c167
< linuxefi /boot/vmlinuz-4.12.14-197.7-default root=UUID=923d921d-0b7d-4cde-863f-c180d07ba58c ${extra_cmdline} splash=silent resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto quiet
---
> linuxefi /boot/vmlinuz-4.12.14-197.7-default root=UUID=923d921d-0b7d-4cde-863f-c180d07ba58c ${extra_cmdline} resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto console=tty0 console=ttyS0,115200n8
197c199
< linuxefi /boot/vmlinuz-4.12.14-195-default root=UUID=923d921d-0b7d-4cde-863f-c180d07ba58c ${extra_cmdline} splash=silent resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto quiet
---
> linuxefi /boot/vmlinuz-4.12.14-195-default root=UUID=923d921d-0b7d-4cde-863f-c180d07ba58c ${extra_cmdline} resume=/dev/disk/by-path/pci-0000:04:00.0-part4 mitigations=auto console=tty0 console=ttyS0,115200n8
sles-151-efi-srv01:~ #
Once here we are ready to reboot the system.
Accessing the guest via the serial console
If your sles 15 guest system was correctly configured as stated above, you should be able now to connect to it using any software that work on a serial connections like minicom, screen, or even putty.
In this demonstration, I will be accessing the kvm guest using the virsh console command from the KVM host.
Let's find the sles 15 guest and ID first:
[root@casld002 ~]# virsh -c qemu:///system list
Id Name State
----------------------------------------
2 fedora-gw running
3 centOS-7.2-iscsitarget running
4 ubuntu-16.4-dns01 running
5 ubunser-14.4-nfssrv running
8 ubunser-14.4-ldap1srv running
10 srv01-opensuse151 running
11 sles-15.1-efi-srv01 running
[root@casld002 ~]#
And now let's find out the serial console device created in the KVM host for the guest of our interest:
[root@casld002 ~]# virsh -c qemu:///system ttyconsole sles-15.1-efi-srv01
/dev/pts/7
[root@casld002 ~]#
[root@casld002 ~]# virsh -c qemu:///system console sles-15.1-efi-srv01
Connected to domain sles-15.1-efi-srv01
Escape character is ^]
sles-151-efi-srv01 login: jesus
Password:
Last login: Mon Jun 8 14:41:18 from 192.168.100.1
jesus@sles-151-efi-srv01:~>
Or even using the screen program launched also from the kvm host
[root@casld002 ~]# screen /dev/pts/7 115200
You can detach from the serial console session doing Ctrl+] or Ctrl+A :quit (if using screen)
Using effectively the serial console to debug booting problems:
Let's say that you are having booting issues and you would like to have in a text file stored in the KVM host file system all the output messages generated on the local console while booting the guest so that you can inspect them later at your own pace or to send the file to support:
[ OK ] Started Load AppArmor profiles.
[ TIME ] Timed out waiting for device dev-di…ackages\x2dx86_64\x2dBuild.device.
[DEPEND] Dependency failed for /media/sles15sp1-pack.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Early Kernel Boot Messages.
Starting Restore /run/initramfs on shutdown...
[ OK ] Reached target User and Group Name Lookups.
[ OK ] Reached target Timers.
[ OK ] Closed Syslog Socket.
[ OK ] Reached target Network (Pre).
[ OK ] Reached target Host and Network Name Lookups.
[ OK ] Reached target System Time Synchronized.
[ OK ] Reached target Remote File Systems.
[ OK ] Reached target Login Prompts.
[ OK ] Reached target Network.
[ OK ] Reached target Network is Online.
[ OK ] Reached target Sockets.
[ OK ] Reached target Paths.
Starting Tell Plymouth To Write Out Runtime Data...
Starting Create Volatile Files and Directories...
[ OK ] Started Emergency Shell.
[ OK ] Reached target Emergency Mode.
[ OK ] Started Restore /run/initramfs on shutdown.
[ OK ] Started Create Volatile Files and Directories.
Starting Security Auditing Service...
[ OK ] Started Tell Plymouth To Write Out Runtime Data.
[ OK ] Started Security Auditing Service.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Started Update UTMP about System Boot/Shutdown.
Starting Update UTMP about System Runlevel Changes...
[ OK ] Started Update UTMP about System Runlevel Changes.
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to boot
into default mode.
Give root password for maintenance
(or press Control-D to continue):
How do you manage to trap all the output to a file via the serial console?
Very easy:
From a different shell session connect to the guest using the serial console, but now make use of the "tee" tool to duplicate the output to a text file:
[root@casld002 ~]# virsh -c qemu:///system console sles-15.1-efi-srv01 | tee -a /tmp/sles-151-srv01.console.out
Now reboot the guest and you will end with a file having all the messages written to the file of your liking:
[root@casld002 ~]# ls -lrt /tmp/sles-151-srv01.console.*
-rw-r--r--. 1 root root 268010 Jun 8 18:55 /tmp/sles-151-srv01.console.20200608-18411591634467.out
[root@casld002 ~]#
Additional Information
The procedure described above is also valid for sles 12 based OSes.
For KVM guests running on ppc64le systems the procedure is still valid, but you have to keep in mind that the serial interfaces and devices used could be different.
Finally, the procedure stated above assumed that the hardware for kvm guest did not support serial devices and that the configuration we wanted to get was a persistent one.
While this is usually the case, a more direct and quick way to proceed when trying to do very specific and sporadic debugging tasks is to operate directly the boot loader stanzas at boot time and adding manually the required kernel parameters for the serial device for a one time troubleshooting test. Of course, that will work if the kvm guest has been attached previously the corresponding controller definitions in the device section of the xml file.
Document Location
Worldwide
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SGMV168","label":"SUSE Linux Enterprise Server"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB57","label":"Power"}}]
Was this topic helpful?
Document Information
Modified date:
01 April 2021
UID
ibm16218944