GNU Project Debugger: More fun with GDB

Customizing the debugger

The previous article on this topic, "Fun with strace and GDB," covered the basics of using these tools to explore your system and attach to programs that are already running to see what they were doing. This article presses on to customize the debugger to make the experience more personal and efficient.

When GDB, the GNU Project Debugger, starts up, it looks for a file in the current user's home directory called .gdbinit; if the file exists, GDB executes all the commands in the file. Normally, this file is used for simple configuration commands, such as setting the default assembler format desired (Intel® or Motorola) or default radix to display input and output data (decimal or hexadecimal). It can also read a macro-coding language that allows more powerful customizations. The language follows this basic format:

define <command>
document <command>
<help text>

The command is known as a user command. All other standard GDB commands can be combined with flow-control instructions and passed parameters to create a language that allows you to customize the behavior of the debugger for the specific application being debugged.

Start simple: Clear the screen

It's always good to start simple and build up from there. Start the xterm, fire up your favorite editor, and let's begin creating a useful .gdbinit file! The output from debuggers can be messy and, as a matter of personal preference, many people like to be able to clear the screen when working with any tool that has the potential for creating cluttering. GDB has no built-in command for clearing the screen, but it can call shell functions; the following code steps outside the debugger to clear the xterm console with the cls command:

define cls
shell clear
document cls
Clears the screen with a simple command.

The upper portion of the definition, bounded by the define ... end verbs, frames the code that is executed when the command is invoked.

The lower portion of the definition, bounded by document ... end, is used by the GDB command interpreter to display help text associated with the cls command when you type help cls.

After you type the code into the .gdbinit file, fire up GDB and enter the cls command. Your screen clears, and all you see is the GDB prompt. Your journey into GDB customization has begun!

The importance of documentation

If you enter the help user command, you see a summary of all the user commands you've entered in the .gdbinit file. The designers of the .gdbinit user-defined commands have provided an important feature that you shouldn't ignore when you're writing your own commands: the document ... end clause. Maintaining functional documentation of how commands work becomes critical as the number of these commands increase.

You might have already experienced the problem. Suppose you wrote some code from a few years ago; then, when you revisit it, perhaps to fix a bug or modify it by adding a new feature, you realize that you're having a hard time understanding your own code. Good programmers make it a habit to keep code short, simple, and well-documented so that it's maintainable.

What's true for programming code in general is true for debugger code. Keeping careful notes and well-documented code will serve you well as you work your way through this most rewarding of careers.

Community uses for GDB

Human beings learn new things many ways, including by studying what others have done. Budding automotive engineers start by opening the hood of their first car, pulling out their tools, and removing parts to clean and study. Such an exercise lets them keep a clean machine while learning how the automobile engine works.

Budding computer scientists are no different in that they want to see just how programs work -- how they interact with dynamic libraries and the native operating system. The tool for seeing how these things work is the debugger. Computer programming is a complex activity, and by being able to interact with a community of like-minded people, asking questions and getting answers, new computer scientists can satisfy their need for knowledge.

In the global programming community, there are always a healthy number of people who hunger and thirst for knowledge. They aren't content with just running programs on their computers -- they want to know more. They want to know how those programs run, and they enjoy exploring the functionality of their systems with the best tool available for the purpose: the debugger. By reverse engineering, a way of learning how programs work by running them under a debugger and keeping careful notes of how they do what they do, you can learn a lot from what the authors of the program being studied have done. Many low-level details involved in programming are undocumented; the only way to learn about them is to see them in action.

Reverse engineering has an undeserved reputation as a black art practiced only by hackers and criminals intent on breaking copy-protection systems and writing worms and viruses to inflict harm on the world of computers. Although there are such people, the vast majority of those who study how programs work, using debuggers and reverse engineering, are the present and future software engineers who want and need to know how these things work. They have formed online communities to share their knowledge and discoveries; discouraging this activity is destructive and retards the future progress of computer science.

Many of the user functions defined in this article come from such communities of hungry knowledge seekers. If you want to know more about them, you're encouraged to study the Web sites referenced in the Related topics section of this document.

Breakpoint aliases

It's a well-known fact that many of the GDB commands are too verbose. Even though they can be abbreviated, the GDB macro language allows for even greater simplifications. Commands, such as info breakpoints, can be made as simple as bpl. Listing 1 shows a great set of such simple and highly useful breakpoint alias user commands to edit into your growing .gdbinit file.

Listing 1: Breakpoint alias commands
define bpl
info breakpoints
document bpl
List breakpoints

define bp
break * $arg0
document bp
Set a breakpoint on address
Usage: bp addr

define bpc
clear $arg0
document bpc
Clear breakpoint at function/address
Usage: bpc addr

define bpe
enable $arg0
document bpe
Enable breakpoint #
Usage: bpe num

define bpd
disable $arg0
document bpd
Disable breakpoint #
Usage: bpd num

define bpt
tbreak $arg0
document bpt
Set a temporary breakpoint on address
Usage: bpt addr

define bpm
awatch $arg0
document bpm
Set a read/write breakpoint on address
Usage: bpm addr

Once you get used to using breakpoint alias commands, your debugging sessions become more rewarding; these commands greatly increase the debugger's effectiveness, because you can accomplish more with less effort.

Displaying process information

The GDB user-defined commands can be called by other user-defined commands, leading to greater effectiveness for them all. This is the incremental nature of programming languages -- writing lower-level functions that are called by progressively higher-level functions until the tools conveniently do what you want them to do with minimal effort on your part. The next set of GDB definitions to incorporate into your .gdbinit file display useful process information when they're invoked, as shown in Listing 2.

Listing 2: Process information commands
define argv
show args
document argv
Print program arguments

define stack
info stack
document stack
Print call stack

define frame
info frame
info args
info locals
document frame
Print stack frame

define flags
if (($eflags >> 0xB) & 1 )
printf "O "
printf "o "
if (($eflags >> 0xA) & 1 )
printf "D "
printf "d "
if (($eflags >> 9) & 1 )
printf "I "
printf "i "
if (($eflags >> 8) & 1 )
printf "T "
printf "t "
if (($eflags >> 7) & 1 )
printf "S "
printf "s "
if (($eflags >> 6) & 1 )
printf "Z "
printf "z "
if (($eflags >> 4) & 1 )
printf "A "
printf "a "
if (($eflags >> 2) & 1 )
printf "P "
printf "p "
if ($eflags & 1)
printf "C "
printf "c "
printf "\n"
document flags
Print flags register

define eflags
printf "     OF <%d>  DF <%d>  IF <%d>  TF <%d>",\
        (($eflags >> 0xB) & 1 ), (($eflags >> 0xA) & 1 ), \
        (($eflags >> 9) & 1 ), (($eflags >> 8) & 1 )
printf "  SF <%d>  ZF <%d>  AF <%d>  PF <%d>  CF <%d>\n",\
        (($eflags >> 7) & 1 ), (($eflags >> 6) & 1 ),\
        (($eflags >> 4) & 1 ), (($eflags >> 2) & 1 ), ($eflags & 1)
printf "     ID <%d>  VIP <%d> VIF <%d> AC <%d>",\
        (($eflags >> 0x15) & 1 ), (($eflags >> 0x14) & 1 ), \
        (($eflags >> 0x13) & 1 ), (($eflags >> 0x12) & 1 )
printf "  VM <%d>  RF <%d>  NT <%d>  IOPL <%d>\n",\
        (($eflags >> 0x11) & 1 ), (($eflags >> 0x10) & 1 ),\
        (($eflags >> 0xE) & 1 ), (($eflags >> 0xC) & 3 )
document eflags
Print entire eflags register

define reg
printf "     eax:%08X ebx:%08X  ecx:%08X ",  $eax, $ebx, $ecx
printf " edx:%08X     eflags:%08X\n",  $edx, $eflags
printf "     esi:%08X edi:%08X  esp:%08X ",  $esi, $edi, $esp
printf " ebp:%08X     eip:%08X\n", $ebp, $eip
printf "     cs:%04X  ds:%04X  es:%04X", $cs, $ds, $es
printf "  fs:%04X  gs:%04X  ss:%04X    ", $fs, $gs, $ss
document reg
Print CPU registers

define func
info functions
document func
Print functions in target

define var
info variables
document var
Print variables (symbols) in target

define lib
info sharedlibrary
document lib
Print shared libraries linked to target

define sig
info signals
document sig
Print signal actions for target

define thread
info threads
document thread
Print threads in target

define u
info udot
document u
Print kernel 'user' struct for target

define dis
disassemble $arg0
document dis
Disassemble address
Usage: dis addr

Hexadecimal and ASCII dumping commands

The next set of definitions to incorporate into the .gdbinit file includes enhanced hexadecimal and ASCII dump functions, as shown in Listing 3. Programmers take note: If you want to create great software, then add the ability to program macros, allowing your community of users to be able to enhance your tool to suit their own tastes. GDB is great software!

Listing 3: Hexadecimal and ASCII dumping commands
define ascii_char
set $_c=*(unsigned char *)($arg0)
if ( $_c < 0x20 || $_c > 0x7E )
printf "."
printf "%c", $_c
document ascii_char
Print the ASCII value of arg0 or '.' if value is unprintable

define hex_quad
printf "%02X %02X %02X %02X  %02X %02X %02X %02X",                          \
               *(unsigned char*)($arg0), *(unsigned char*)($arg0 + 1),      \
               *(unsigned char*)($arg0 + 2), *(unsigned char*)($arg0 + 3),  \
               *(unsigned char*)($arg0 + 4), *(unsigned char*)($arg0 + 5),  \
               *(unsigned char*)($arg0 + 6), *(unsigned char*)($arg0 + 7)
document hex_quad
Print eight hexadecimal bytes starting at arg0

define hexdump
printf "%08X : ", $arg0
hex_quad $arg0
printf " - "
hex_quad ($arg0+8)
printf " "

ascii_char ($arg0)
ascii_char ($arg0+1)
ascii_char ($arg0+2)
ascii_char ($arg0+3)
ascii_char ($arg0+4)
ascii_char ($arg0+5)
ascii_char ($arg0+6)
ascii_char ($arg0+7)
ascii_char ($arg0+8)
ascii_char ($arg0+9)
ascii_char ($arg0+0xA)
ascii_char ($arg0+0xB)
ascii_char ($arg0+0xC)
ascii_char ($arg0+0xD)
ascii_char ($arg0+0xE)
ascii_char ($arg0+0xF)

printf "\n"
document hexdump
Display a 16-byte hex/ASCII dump of arg0

define ddump
printf "[%04X:%08X]------------------------", $ds, $data_addr
printf "---------------------------------[ data]\n"
set $_count=0
while ( $_count < $arg0 )
set $_i=($_count*0x10)
hexdump ($data_addr+$_i)
set $_count++
document ddump
Display $arg0 lines of hexdump for address $data_addr

define dd
if ( ($arg0 & 0x40000000) || ($arg0 & 0x08000000) || ($arg0 & 0xBF000000) )
set $data_addr=$arg0
ddump 0x10
printf "Invalid address: %08X\n", $arg0
document dd
Display 16 lines of a hex dump for $arg0

define datawin
if ( ($esi & 0x40000000) || ($esi & 0x08000000) || ($esi & 0xBF000000) )
set $data_addr=$esi
if ( ($edi & 0x40000000) || ($edi & 0x08000000) || ($edi & 0xBF000000) )
set $data_addr=$edi
if ( ($eax & 0x40000000) || ($eax & 0x08000000) || \
      ($eax & 0xBF000000) )
set $data_addr=$eax
set $data_addr=$esp
 ddump 2
document datawin
Display esi, edi, eax, or esp in the data window

Process context commands

Finally, when you're debugging a running process, it's often necessary to get an overall view of the process context. The useful process context commands in Listing 4 are built by using the previously defined data dumping functions.

Listing 4: Process context commands
define context
printf "_______________________________________"
printf "________________________________________\n"
printf "[%04X:%08X]------------------------", $ss, $esp
printf "---------------------------------[stack]\n"
hexdump $sp+0x30
hexdump $sp+0x20
hexdump $sp+0x10
hexdump $sp
printf "[%04X:%08X]------------------------", $cs, $eip
printf "---------------------------------[ code]\n"
x /6i $pc
printf "---------------------------------------"
printf "---------------------------------------\n"
document context
Print regs, stack, ds:esi, and disassemble cs:eip

define context-on
document context-on
Enable display of context on every program stop

define context-off
document context-on
Disable display of context on every program stop

# Calls "context" at every breakpoint.
define hook-stop

# Init parameters
set output-radix 0x10
set input-radix 0x10
set disassembly-flavor intel

hook-stop is a special definition that GDB calls at every breakpoint event. In this case, the context listing is generated so that you can clearly see the effect of each instruction being executed by the processor.

A debugging session with the new functionality

Let's experiment with your new set of tools to see how they work when debugging an old friend, the nweb server code by IBM developerWorks contributing writer Nigel Griffiths. (See the Related topics section for a link to Nigel's article "nweb: a tiny, safe Web server (static pages only).")

After downloading the file to your $HOME/downloads directory, type the following commands to extract, compile, and launch nweb. (Please note that this assumes you're compiling the program to a Linux® workstation with the Intel Pentium as the central processing unit (CPU) -- the .gdbinit code was written for the Intel Pentium-type processor and compatibles only.)

$ cd src
$ mkdir nweb
$ cd nweb
$ unzip $HOME/downloads/
$ gcc -ggdb -O -DLINUX nweb.c -o nweb
$ ./nweb 9090 $HOME/src/nweb &

Note: The -ggdb option in this example differs from Nigel's article in that it tells the GNU Compiler Collection (GCC) to optimize the program for debugging with GDB.

Next, to verify that the nweb server is running, use the ps command to check it:

$ ps
  PID TTY          TIME CMD
 2913 pts/5    00:00:00 bash
 4009 pts/5    00:00:00 nweb
 4011 pts/5    00:00:00 ps

Finally, launch a Web browser on your computer and type http://localhost:9090 in the address bar.

Next, start GDB and, as before, attach to the currently running nweb instance, as shown in Listing 5.

Listing 5: Launch GDB
$ gdb --quiet
(gdb) attach 4009
Attaching to process 4009
Reading symbols from /home/bill/src/nweb/nweb...done.
Reading symbols from /lib/tls/
Loaded symbols for /lib/tls/
Reading symbols from /lib/
Loaded symbols for /lib/
     eax:FFFFFE00 ebx:00000005  ecx:BFFFF680  edx:00000001     eflags:00000246
     esi:00000005 edi:00000000  esp:BFFFF66C  ebp:BFFFF6A8     eip:FFFFE410
     cs:0073  ds:007B  es:007B  fs:0000  gs:0033  ss:007B    o d I t s Z a P c
BFFFF69C : 14 0A 13 42  60 53 01 40 - 24 8F 04 08  C8 F6 FF BF ...B`S.@$.......
BFFFF68C : A6 8E 04 08  14 0A 13 42 - 70 C6 00 40  10 00 00 00 .......Bp..@....
BFFFF67C : 82 8E 04 08  00 00 00 00 - C4 C6 04 08  98 F6 FF BF ................
BFFFF66C : A8 F6 FF BF  01 00 00 00 - 80 F6 FF BF  81 EA 0D 42 ...............B
[007B:FFFFFE00]---------------------------------------------------------[ data]
FFFFFE00 : Error while running hook_stop:
Cannot access memory at address 0xfffffe00
0xffffe410 in ?? ()

The -quiet option tells the GDB Debugger to display only its prompt and not all the other startup information that it typically displays. If you want the extra text, leave the -quiet option off.

The attach 4009 command starts debugging the currently running nweb server, and the GDB Debugger responds in kind by reading all the symbolic information about the process it can.

You'll notice that the context code ran and displayed a lot of useful information about the current process, but it was unable to access memory in the data segment. This is not a serious problem, and it should be ignored. Sometimes, the protection scheme of a protected mode processor won't let you see all that you might want to see. In this case, it isn't important.

Next, use the info command to list information about the program you're exploring (see Listing 6).

Listing 6: The info command lists program information
(gdb) info proc
process 4009
cmdline = './nweb'
cwd = '/home/bill/src/nweb'
exe = '/home/bill/src/nweb/nweb'

Watching it live

Because you're watching an actual running program, you can set breakpoints and then see the program as it replies to browser requests and transmits .html and .jpg files to the browser making the request. Listing 7 shows how you can do so.

Listing 7: Set breakpoints
(gdb) b 188
Breakpoint 1 at 0x8048e70: file nweb.c, line 188.
(gdb) commands 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
(gdb) c

At this point, the GDB Debugger is set to break at the line where the nweb server accepts browser requests; the debugger will simply display the request and continue processing other requests without interrupting the running program. Refresh the http://localhost:9090/ page in your browser a few times, and watch the GDB Debugger display the breakpoint and continue running.

While refreshing the browser page, you should see breakpoint information, such as that shown in Listing 8, scrolling in the GDB Debugger xterm. You can stop debugging in the nweb server by pressing Ctrl+C. After stopping the tracing, you can exit the GDB Debugger by typing the quit command.

Listing 8: Breakpoint information in the GDB Debugger xterm
     eax:00000000 ebx:00000001  ecx:00000000  edx:00000001     eflags:00000206
     esi:00000006 edi:00000000  esp:BFFFF690  ebp:BFFFF6A8     eip:08048E70
     cs:0073  ds:007B  es:007B  fs:0000  gs:0033  ss:007B    o d I t s z a P c
BFFFF6C0 : 03 00 00 00  D4 86 04 08 - 00 00 00 00  F5 86 04 08 ................
BFFFF6B0 : 03 00 00 00  F4 F6 FF BF - 04 F7 FF BF  2C 58 01 40 ............,X.@
BFFFF6A0 : 60 53 01 40  24 8F 04 08 - C8 F6 FF BF  04 55 01 42 `S.@$........U.B
BFFFF690 : 14 0A 13 42  70 C6 00 40 - 10 00 00 00  14 0A 13 42 ...Bp..@.......B
[007B:BFFFF690]---------------------------------------------------------[ data]
BFFFF690 : 14 0A 13 42  70 C6 00 40 - 10 00 00 00  14 0A 13 42 ...Bp..@.......B
BFFFF6A0 : 60 53 01 40  24 8F 04 08 - C8 F6 FF BF  04 55 01 42 `S.@$........U.B
[0073:08048E70]---------------------------------------------------------[ code]
0x8048e70 <main+718>:   sub    esp,0x4
0x8048e73 <main+721>:   lea    eax,[ebp-16]
0x8048e76 <main+724>:   push   eax
0x8048e77 <main+725>:   push   0x804c6c4
0x8048e7c <main+730>:   push   edi
0x8048e7d <main+731>:   call   0x80485e4 <accept>
Breakpoint 1, main (argc=3, argv=0x1) at nweb.c:188
188           if((socketfd = accept(listenfd, (struct sockaddr *)&cli_addr, &length)) < 0)
Program received signal SIGINT, Interrupt.
0xffffe410 in ?? ()
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /home/bill/src/nweb/nweb, process 4009

As you can see, the information displayed by the context function is far more detailed than you normally see with the default GDB hook_stop vector. (You'll also notice that you can now access your data segment, too.) With these GDB enhancements, you can see the exact state of the CPU every time the breakpoint is reached and with every step operation you execute. Stepping through each command and watching how the register and memory values are affected is also a great way to learn the basics of the Intel machine language commands.

As is true for all programs, the code in the .gdbinit file provides endless opportunities for enhancement and improvement. By all means, this isn't the end! You're highly encouraged to use the commands described here and to add many more to the growing set of .gdbinit customizations. As you explore and use the tools, share them with the broader community so everyone grows in knowledge.


More people should take up the deep study of how programming tools work, come out of their shells, and contribute to the overall communities of those who seek such knowledge. Visit some of these online communities, and even consider forming your own so that the technological innovations of the future will come more quickly into fruition.

Downloadable resources

Related topics

Zone=AIX and UNIX
ArticleTitle=GNU Project Debugger: More fun with GDB