Level: Intermediate Jayesh Rane (jayesh.rane@in.ibm.com), Systems Software Engineer, IBM
06 Mar 2007 Explore how to remotely debug a FreeBSD kernel that is running on a target machine without affecting system performance. In this article, examine setting up the debug environment using serial communication port, compiling modified kernel code, debugging, and troubleshooting tips.
Introduction
FreeBSD is an advanced operating system derived from BSD, the version of UNIX® developed at the University of California, Berkeley. It provides high performance, greater compatibility with other operating systems, and less system administration. FreeBSD includes kernel support for stateful IP firewalling, and other services such as IP proxy gateways, access control lists, mandatory access control, jail-based virtual hosting, and cryptographically protected storage.
When developing a kernel code or running a development kernel, sometimes a kernel panics. It is very important to diagnose the core dump to extract useful information. If you try to use the same development machine for debugging this core dump, it might affect system performance. You can debug only one core dump at a time. To debug your kernel core dump efficiently, it is best to set up a remote debugging session between the development machine and target machine, and remotely debug the kernel using serial communication.
The kgdb utility is a debugger based on gdb that allows debugging of kernel core files. FreeBSD 5.3 and later come with the kgdb utility. If you have an older version of FreeBSD, you can still use gdb -k to read a core dump file. kgdb helps to place breakpoints in kernel code, so you can step through it and observe the variables.
This article shows you how to set up the debug environment using serial communication port, how to compile modified kernel code, and how to start kernel debugging. Some troubleshooting tips are also included.
Setting up the debugging environment
To set up the remote debugging environment for FreeBSD kernel debugging, you need two machines: a target machine and a development machine. The target machine runs with the new kernel image that needs to be debugged. The development machine is used to debug the kernel running on the target machine. For better performance, you can use the machine with the better configuration as the development machine.
Null modem cable
Null modem is a communication method to connect two computers directly using a RS-232 serial cable. The original RS-232 standard only defined the connection of computers with modems. With a null modem connection, the transmit and receive lines are cross-linked. Depending on the purpose, sometimes one or more handshake lines are cross-linked. Several wiring layouts are in use, because the null modem connection is not covered by a standard.
If you connect the null modem cable to a serial port, the debugger can run at its current top speed of 19200, as compared to the speed of 9600 over a modem. This configuration is generally considered more reliable than using a modem because:
- A null modem cable is a serial cable that allows data to be sent between two serial ports. Typically, you connect a serial port to a serial device. In this case, you must configure the target computer to accept a serial-port-to-serial-port connection. It is important that you use a null modem cable; if you use a typical serial cable, the target computer expects a serial device and not another serial port, and you cannot debug the computer.
- When you use a null modem to make a direct serial connection between two computers, your cable must be wired, as shown in Listing 1 below.
Listing 1. 25-Pin NULL modem cabling
2 3 Transmit Data
3 2 Receive Data
4 5 Request to Send
5 4 Clear to Send
6 20 Data Set Ready and Carrier Detect
7 7 Signal Ground
20 6 Data Terminal Ready
|
Modifying kernel files
You need to modify certain kernel configuration files in the FreeBSD kernel to make it aware of the remote kernel debugging environment that is using serial communication. First, make a backup of the original /usr/src/sys/i386/conf/GENERIC configuration file, as follows:
$cd /usr/src/sys/i386/conf/
$cp GENERIC MYKERNEL |
On your development machine, open the configuration file in your favorite editor (vi or emacs):
$ emacs /usr/src/sys/i386/conf/MYKERNEL |
Add the following lines, as follows:
options ddb
#This line will add the debugger into the kernel after the kernel compilation.
makeoptions debug = -g
#This line will tell the compiler to add debugging information into the new compiled
#kernel image, which you are going to put on target machine. |
To activate the serial port communication for kernel debugging, you need to modify the same kernel configuration file. In the /usr/src/sys/i386/conf/MYKERNEL file, locate the sio0 word, which refers to your COM1 serial port on your computer. Set the 0x80 port flag, as follows:
device sio0 at isa? port IO_COM1 flags 0x80 |
Compiling the FreeBSD kernel
Once you're done with the changes to the kernel configuration files, you have to compile your kernel so that the new kernel image can be debugged.
Configure the kernel using the following command:
572
$ cd /usr/src/sys/i386/conf/
$ config -g MYKERNEL |
The -g option adds debugging information into the kernel image. After the successful run of the above command, notice how it creates the MYKERNEL directory under /usr/src/sys/compile/.
Change the directory to /usr/src/sys/compile/MYKERNEL/, as follows:
$ cd /usr/src/sys/compile/MYKERNEL/ |
You need to validate the various dependencies among different modules or files by issuing the following command:
At this point, it's time to build the kernel and install the new kernel image, as follows:
After the successful build operation, your kernel image in the boot directory is overwritten by the newly built kernel image; the old kernel image is saved as kernel.old.
Debugging the kernel
Before starting, make sure that you have connected the null modem cable between the development and target machines.
Once the compilation is done, copy the /usr/src/sys/compile/MYKERNEL/kernel.debug kernel image from the development machine to the target machine. Because you need to debug the new kernel.debug image, you have to reboot the target machine with the kernel.debug image.
Issue the following commands when booting the machine to make it go into debugger mode:
load kernel.debug
boot -d |
A debugger interface is started, which allows you to start debugging from the development machine. Enter:
You need to force a next step immediately by simply typing s (step). To start the kernel debugging, open the emacs editor on the development machine. It allows you to open the kgdb utility; only in emacs can you do the kernel debugging line by line. After opening the emacs editor, load gdb with the kernel.debug image that you want to debug, with the -k option for kernel debugging. It shows a kgdb prompt in the emacs editor. To start the remote debug session, enter:
(kgdb) target remote /dev/cuaa0 |
Troubleshooting
It is very important that you have the correct null modem cable with you to set up the debugging environment. This article discusses the 25-pin null modem cabling. If you want to use 9-pin null modem cabling, see the Resources section for more information about null modem setup.
At the time of remote debugging initiation, you might get the Ignoring packet error... error. This error indicates that your development machine and target machine do not have the same baud rate. The baud rate of the machine is the number of symbols per second transferred. The baud rate might not be equal to the bit rate, as a symbol might have more than two states. You need to set both machines with the same baud rate. It can be done using the following command:
Conclusion
There are various advantages to using remote kernel debugging instead of traditional kernel debugging. Your system’s performance is not affected at all, since you are debugging it remotely on a development machine (one at a time). You can debug as many kernel images (which are residing on target machines) as you want to debug; it becomes transparent to users once debugging starts. If gdb dies or terminates because of a problem, start a new instance of gdb and connect to kgdb with a target remote command.
Resources Learn
-
Null modem setup: Learn about using 9-pin null modem cabling.
-
FreeBSD operating system: Read about all the news, events, and resources available for FreeBSD.
-
GDB: The GNU project debugger.
-
kgdb: Learn more about the kernel debugger for FreeBSD.
-
"Installing FreeBSD" (developerWorks, May 2001): Learn how to make FreeBSD run on your computer.
- Search the AIX® and UNIX library by topic:
- AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
- New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
- AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.
- Safari bookstore: Visit this e-reference library to find specific technical resources.
- developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
- Podcasts: Tune in and catch up with IBM technical experts.
Get products and technologies
-
FreeBSD: Download the FreeBSD operating system, applications, and utility software.
- IBM trial software: Build your next development project with software for download directly from developerWorks.
Discuss
-
Participate in the AIX and UNIX forums:
About the author  | 
| | Jayesh V. Rane works as a Systems Software Engineer for IBM and has five years of experience in product development, mostly working on network programming and storage management software. He has worked extensively on FreeBSD and AIX operating systems. Jayesh is currently working with the IBM India Systems and Technology Lab in Pune, India. You can contact him at jayesh.rane@in.ibm.com. |
Rate this page
| 
