The previous article on this topic, "Fun with strace and GDB," covered the basics of using these tools to explore your system and attach to programs that are already running to see what they were doing. This article presses on to customize the debugger to make the experience more personal and efficient.

When GDB, the GNU Project Debugger, starts up, it looks for a file in the current user's home directory called .gdbinit; if the file exists, GDB executes all the commands in the file. Normally, this file is used for simple configuration commands, such as setting the default assembler format desired (Intel® or Motorola) or default radix to display input and output data (decimal or hexadecimal). It can also read a macro-coding language that allows more powerful customizations. The language follows this basic format:

define <command>
<code>
end
document <command>
<help text>
end

The command is known as a user command. All other standard GDB commands can be combined with flow-control instructions and passed parameters to create a language that allows you to customize the behavior of the debugger for the specific application being debugged.

Start simple: Clear the screen

It's always good to start simple and build up from there. Start the xterm, fire up your favorite editor, and let's begin creating a useful .gdbinit file! The output from debuggers can be messy and, as a matter of personal preference, many people like to be able to clear the screen when working with any tool that has the potential for creating cluttering. GDB has no built-in command for clearing the screen, but it can call shell functions; the following code steps outside the debugger to clear the xterm console with the cls command:

define cls
shell clear
end
document cls
Clears the screen with a simple command.
end

The upper portion of the definition, bounded by the define ... end verbs, frames the code that is executed when the command is invoked.

The lower portion of the definition, bounded by document ... end, is used by the GDB command interpreter to display help text associated with the cls command when you type help cls.

After you type the code into the .gdbinit file, fire up GDB and enter the cls command. Your screen clears, and all you see is the GDB prompt. Your journey into GDB customization has begun!

The importance of documentation

If you enter the help user command, you see a summary of all the user commands you've entered in the .gdbinit file. The designers of the .gdbinit user-defined commands have provided an important feature that you shouldn't ignore when you're writing your own commands: the document ... end clause. Maintaining functional documentation of how commands work becomes critical as the number of these commands increase.

You might have already experienced the problem. Suppose you wrote some code from a few years ago; then, when you revisit it, perhaps to fix a bug or modify it by adding a new feature, you realize that you're having a hard time understanding your own code. Good programmers make it a habit to keep code short, simple, and well-documented so that it's maintainable.

What's true for programming code in general is true for debugger code. Keeping careful notes and well-documented code will serve you well as you work your way through this most rewarding of careers.

Community uses for GDB

Human beings learn new things many ways, including by studying what others have done. Budding automotive engineers start by opening the hood of their first car, pulling out their tools, and removing parts to clean and study. Such an exercise lets them keep a clean machine while learning how the automobile engine works.

Budding computer scientists are no different in that they want to see just how programs work -- how they interact with dynamic libraries and the native operating system. The tool for seeing how these things work is the debugger. Computer programming is a complex activity, and by being able to interact with a community of like-minded people, asking questions and getting answers, new computer scientists can satisfy their need for knowledge.

In the global programming community, there are always a healthy number of people who hunger and thirst for knowledge. They aren't content with just running programs on their computers -- they want to know more. They want to know how those programs run, and they enjoy exploring the functionality of their systems with the best tool available for the purpose: the debugger. By reverse engineering, a way of learning how programs work by running them under a debugger and keeping careful notes of how they do what they do, you can learn a lot from what the authors of the program being studied have done. Many low-level details involved in programming are undocumented; the only way to learn about them is to see them in action.

Reverse engineering has an undeserved reputation as a black art practiced only by hackers and criminals intent on breaking copy-protection systems and writing worms and viruses to inflict harm on the world of computers. Although there are such people, the vast majority of those who study how programs work, using debuggers and reverse engineering, are the present and future software engineers who want and need to know how these things work. They have formed online communities to share their knowledge and discoveries; discouraging this activity is destructive and retards the future progress of computer science.

Many of the user functions defined in this article come from such communities of hungry knowledge seekers. If you want to know more about them, you're encouraged to study the Web sites referenced in the Resources section of this document.

Breakpoint aliases

It's a well-known fact that many of the GDB commands are too verbose. Even though they can be abbreviated, the GDB macro language allows for even greater simplifications. Commands, such as info breakpoints, can be made as simple as bpl. Listing 1 shows a great set of such simple and highly useful breakpoint alias user commands to edit into your growing .gdbinit file.

define bpl
info breakpoints
end
document bpl
List breakpoints
end

define bp
set $SHOW_CONTEXT = 1
break * $arg0
end
document bp
Set a breakpoint on address
Usage: bp addr
end

define bpc
clear $arg0
end
document bpc
Clear breakpoint at function/address
Usage: bpc addr
end

define bpe
enable $arg0
end
document bpe
Enable breakpoint #
Usage: bpe num
end

define bpd
disable $arg0
end
document bpd
Disable breakpoint #
Usage: bpd num
end

define bpt
set $SHOW_CONTEXT = 1
tbreak $arg0
end
document bpt
Set a temporary breakpoint on address
Usage: bpt addr
end

define bpm
set $SHOW_CONTEXT = 1
awatch $arg0
end
document bpm
Set a read/write breakpoint on address
Usage: bpm addr
end

Once you get used to using breakpoint alias commands, your debugging sessions become more rewarding; these commands greatly increase the debugger's effectiveness, because you can accomplish more with less effort.

Displaying process information

The GDB user-defined commands can be called by other user-defined commands, leading to greater effectiveness for them all. This is the incremental nature of programming languages -- writing lower-level functions that are called by progressively higher-level functions until the tools conveniently do what you want them to do with minimal effort on your part. The next set of GDB definitions to incorporate into your .gdbinit file display useful process information when they're invoked, as shown in Listing 2.

define argv
show args
end
document argv
Print program arguments
end

define stack
info stack
end
document stack
Print call stack
end

define frame
info frame
info args
info locals
end
document frame
Print stack frame
end

define flags
if (($eflags >> 0xB) & 1 )
printf "O "
else
printf "o "
end
if (($eflags >> 0xA) & 1 )
printf "D "
else
printf "d "
end
if (($eflags >> 9) & 1 )
printf "I "
else
printf "i "
end
if (($eflags >> 8) & 1 )
printf "T "
else
printf "t "
end
if (($eflags >> 7) & 1 )
printf "S "
else
printf "s "
end
if (($eflags >> 6) & 1 )
printf "Z "
else
printf "z "
end
if (($eflags >> 4) & 1 )
printf "A "
else
printf "a "
end
if (($eflags >> 2) & 1 )
printf "P "
else
printf "p "
end
if ($eflags & 1)
printf "C "
else
printf "c "
end
printf "\n"
end
document flags
Print flags register
end

define eflags
printf "     OF <%d>  DF <%d>  IF <%d>  TF <%d>",\
        (($eflags >> 0xB) & 1 ), (($eflags >> 0xA) & 1 ), \
        (($eflags >> 9) & 1 ), (($eflags >> 8) & 1 )
printf "  SF <%d>  ZF <%d>  AF <%d>  PF <%d>  CF <%d>\n",\
        (($eflags >> 7) & 1 ), (($eflags >> 6) & 1 ),\
        (($eflags >> 4) & 1 ), (($eflags >> 2) & 1 ), ($eflags & 1)
printf "     ID <%d>  VIP <%d> VIF <%d> AC <%d>",\
        (($eflags >> 0x15) & 1 ), (($eflags >> 0x14) & 1 ), \
        (($eflags >> 0x13) & 1 ), (($eflags >> 0x12) & 1 )
printf "  VM <%d>  RF <%d>  NT <%d>  IOPL <%d>\n",\
        (($eflags >> 0x11) & 1 ), (($eflags >> 0x10) & 1 ),\
        (($eflags >> 0xE) & 1 ), (($eflags >> 0xC) & 3 )
end
document eflags
Print entire eflags register
end

define reg
printf "     eax:%08X ebx:%08X  ecx:%08X ",  $eax, $ebx, $ecx
printf " edx:%08X     eflags:%08X\n",  $edx, $eflags
printf "     esi:%08X edi:%08X  esp:%08X ",  $esi, $edi, $esp
printf " ebp:%08X     eip:%08X\n", $ebp, $eip
printf "     cs:%04X  ds:%04X  es:%04X", $cs, $ds, $es
printf "  fs:%04X  gs:%04X  ss:%04X    ", $fs, $gs, $ss
flags
end
document reg
Print CPU registers
end

define func
info functions
end
document func
Print functions in target
end

define var
info variables
end
document var
Print variables (symbols) in target
end

define lib
info sharedlibrary
end
document lib
Print shared libraries linked to target
end

define sig
info signals
end
document sig
Print signal actions for target
end

define thread
info threads
end
document thread
Print threads in target
end

define u
info udot
end
document u
Print kernel 'user' struct for target
end

define dis
disassemble $arg0
end
document dis
Disassemble address
Usage: dis addr
end

Hexadecimal and ASCII dumping commands

The next set of definitions to incorporate into the .gdbinit file includes enhanced hexadecimal and ASCII dump functions, as shown in Listing 3. Programmers take note: If you want to create great software, then add the ability to program macros, allowing your community of users to be able to enhance your tool to suit their own tastes. GDB is great software!

define ascii_char
set $_c=*(unsigned char *)($arg0)
if ( $_c < 0x20 || $_c > 0x7E )
printf "."
else
printf "%c", $_c
end
end
document ascii_char
Print the ASCII value of arg0 or '.' if value is unprintable
end

define hex_quad
printf "%02X %02X %02X %02X  %02X %02X %02X %02X",                          \
               *(unsigned char*)($arg0), *(unsigned char*)($arg0 + 1),      \
               *(unsigned char*)($arg0 + 2), *(unsigned char*)($arg0 + 3),  \
               *(unsigned char*)($arg0 + 4), *(unsigned char*)($arg0 + 5),  \
               *(unsigned char*)($arg0 + 6), *(unsigned char*)($arg0 + 7)
end
document hex_quad
Print eight hexadecimal bytes starting at arg0
end

define hexdump
printf "%08X : ", $arg0
hex_quad $arg0
printf " - "
hex_quad ($arg0+8)
printf " "

ascii_char ($arg0)
ascii_char ($arg0+1)
ascii_char ($arg0+2)
ascii_char ($arg0+3)
ascii_char ($arg0+4)
ascii_char ($arg0+5)
ascii_char ($arg0+6)
ascii_char ($arg0+7)
ascii_char ($arg0+8)
ascii_char ($arg0+9)
ascii_char ($arg0+0xA)
ascii_char ($arg0+0xB)
ascii_char ($arg0+0xC)
ascii_char ($arg0+0xD)
ascii_char ($arg0+0xE)
ascii_char ($arg0+0xF)

printf "\n"
end
document hexdump
Display a 16-byte hex/ASCII dump of arg0
end

define ddump
printf "[%04X:%08X]------------------------", $ds, $data_addr
printf "---------------------------------[ data]\n"
set $_count=0
while ( $_count < $arg0 )
set $_i=($_count*0x10)
hexdump ($data_addr+$_i)
set $_count++
end
end
document ddump
Display $arg0 lines of hexdump for address $data_addr
end

define dd
if ( ($arg0 & 0x40000000) || ($arg0 & 0x08000000) || ($arg0 & 0xBF000000) )
set $data_addr=$arg0
ddump 0x10
else
printf "Invalid address: %08X\n", $arg0
end
end
document dd
Display 16 lines of a hex dump for $arg0
end

define datawin
if ( ($esi & 0x40000000) || ($esi & 0x08000000) || ($esi & 0xBF000000) )
set $data_addr=$esi
else
if ( ($edi & 0x40000000) || ($edi & 0x08000000) || ($edi & 0xBF000000) )
set $data_addr=$edi
else
if ( ($eax & 0x40000000) || ($eax & 0x08000000) || \
      ($eax & 0xBF000000) )
set $data_addr=$eax
else
set $data_addr=$esp
end
end
end
 ddump 2
end
document datawin
Display esi, edi, eax, or esp in the data window
end

Process context commands

Finally, when you're debugging a running process, it's often necessary to get an overall view of the process context. The useful process context commands in Listing 4 are built by using the previously defined data dumping functions.

define context
printf "_______________________________________"
printf "________________________________________\n"
reg
printf "[%04X:%08X]------------------------", $ss, $esp
printf "---------------------------------[stack]\n"
hexdump $sp+0x30
hexdump $sp+0x20
hexdump $sp+0x10
hexdump $sp
datawin
printf "[%04X:%08X]------------------------", $cs, $eip
printf "---------------------------------[ code]\n"
x /6i $pc
printf "---------------------------------------"
printf "---------------------------------------\n"
end
document context
Print regs, stack, ds:esi, and disassemble cs:eip
end

define context-on
set $SHOW_CONTEXT = 1
end
document context-on
Enable display of context on every program stop
end

define context-off
set $SHOW_CONTEXT = 1
end
document context-on
Disable display of context on every program stop
end

# Calls "context" at every breakpoint.
define hook-stop
  context
end

# Init parameters
set output-radix 0x10
set input-radix 0x10
set disassembly-flavor intel

hook-stop is a special definition that GDB calls at every breakpoint event. In this case, the context listing is generated so that you can clearly see the effect of each instruction being executed by the processor.

A debugging session with the new functionality

Let's experiment with your new set of tools to see how they work when debugging an old friend, the nweb server code by IBM developerWorks contributing writer Nigel Griffiths. (See the Resources section for a link to Nigel's article "nweb: a tiny, safe Web server (static pages only).")

After downloading the es-nweb.zip file to your $HOME/downloads directory, type the following commands to extract, compile, and launch nweb. (Please note that this assumes you're compiling the program to a Linux® workstation with the Intel Pentium as the central processing unit (CPU) -- the .gdbinit code was written for the Intel Pentium-type processor and compatibles only.)

$ cd src
$ mkdir nweb
$ cd nweb
$ unzip $HOME/downloads/es-nweb.zip
$ gcc -ggdb -O -DLINUX nweb.c -o nweb
$ ./nweb 9090 $HOME/src/nweb &

Note: The -ggdb option in this example differs from Nigel's article in that it tells the GNU Compiler Collection (GCC) to optimize the program for debugging with GDB.

Next, to verify that the nweb server is running, use the ps command to check it:

$ ps
  PID TTY          TIME CMD
 2913 pts/5    00:00:00 bash
 4009 pts/5    00:00:00 nweb
 4011 pts/5    00:00:00 ps

Finally, launch a Web browser on your computer and type http://localhost:9090 in the address bar.

Next, start GDB and, as before, attach to the currently running nweb instance, as shown in Listing 5.

$ gdb --quiet
(gdb) attach 4009
Attaching to process 4009
Reading symbols from /home/bill/src/nweb/nweb...done.
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
_______________________________________________________________________________
     eax:FFFFFE00 ebx:00000005  ecx:BFFFF680  edx:00000001     eflags:00000246
     esi:00000005 edi:00000000  esp:BFFFF66C  ebp:BFFFF6A8     eip:FFFFE410
     cs:0073  ds:007B  es:007B  fs:0000  gs:0033  ss:007B    o d I t s Z a P c
[007B:BFFFF66C]---------------------------------------------------------[stack]
BFFFF69C : 14 0A 13 42  60 53 01 40 - 24 8F 04 08  C8 F6 FF BF ...B`S.@$.......
BFFFF68C : A6 8E 04 08  14 0A 13 42 - 70 C6 00 40  10 00 00 00 .......Bp..@....
BFFFF67C : 82 8E 04 08  00 00 00 00 - C4 C6 04 08  98 F6 FF BF ................
BFFFF66C : A8 F6 FF BF  01 00 00 00 - 80 F6 FF BF  81 EA 0D 42 ...............B
[007B:FFFFFE00]---------------------------------------------------------[ data]
FFFFFE00 : Error while running hook_stop:
Cannot access memory at address 0xfffffe00
0xffffe410 in ?? ()
(gdb)

The -quiet option tells the GDB Debugger to display only its prompt and not all the other startup information that it typically displays. If you want the extra text, leave the -quiet option off.

The attach 4009 command starts debugging the currently running nweb server, and the GDB Debugger responds in kind by reading all the symbolic information about the process it can.

You'll notice that the context code ran and displayed a lot of useful information about the current process, but it was unable to access memory in the data segment. This is not a serious problem, and it should be ignored. Sometimes, the protection scheme of a protected mode processor won't let you see all that you might want to see. In this case, it isn't important.

Next, use the info command to list information about the program you're exploring (see Listing 6).

(gdb) info proc
process 4009
cmdline = './nweb'
cwd = '/home/bill/src/nweb'
exe = '/home/bill/src/nweb/nweb'
(gdb)

Watching it live

Because you're watching an actual running program, you can set breakpoints and then see the program as it replies to browser requests and transmits .html and .jpg files to the browser making the request. Listing 7 shows how you can do so.

(gdb) b 188
Breakpoint 1 at 0x8048e70: file nweb.c, line 188.
(gdb) commands 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>continue
>end
(gdb) c
Continuing.

At this point, the GDB Debugger is set to break at the line where the nweb server accepts browser requests; the debugger will simply display the request and continue processing other requests without interrupting the running program. Refresh the http://localhost:9090/ page in your browser a few times, and watch the GDB Debugger display the breakpoint and continue running.

While refreshing the browser page, you should see breakpoint information, such as that shown in Listing 8, scrolling in the GDB Debugger xterm. You can stop debugging in the nweb server by pressing Ctrl+C. After stopping the tracing, you can exit the GDB Debugger by typing the quit command.

_______________________________________________________________________________
     eax:00000000 ebx:00000001  ecx:00000000  edx:00000001     eflags:00000206
     esi:00000006 edi:00000000  esp:BFFFF690  ebp:BFFFF6A8     eip:08048E70
     cs:0073  ds:007B  es:007B  fs:0000  gs:0033  ss:007B    o d I t s z a P c
[007B:BFFFF690]---------------------------------------------------------[stack]
BFFFF6C0 : 03 00 00 00  D4 86 04 08 - 00 00 00 00  F5 86 04 08 ................
BFFFF6B0 : 03 00 00 00  F4 F6 FF BF - 04 F7 FF BF  2C 58 01 40 ............,X.@
BFFFF6A0 : 60 53 01 40  24 8F 04 08 - C8 F6 FF BF  04 55 01 42 `S.@$........U.B
BFFFF690 : 14 0A 13 42  70 C6 00 40 - 10 00 00 00  14 0A 13 42 ...Bp..@.......B
[007B:BFFFF690]---------------------------------------------------------[ data]
BFFFF690 : 14 0A 13 42  70 C6 00 40 - 10 00 00 00  14 0A 13 42 ...Bp..@.......B
BFFFF6A0 : 60 53 01 40  24 8F 04 08 - C8 F6 FF BF  04 55 01 42 `S.@$........U.B
[0073:08048E70]---------------------------------------------------------[ code]
0x8048e70 <main+718>:   sub    esp,0x4
0x8048e73 <main+721>:   lea    eax,[ebp-16]
0x8048e76 <main+724>:   push   eax
0x8048e77 <main+725>:   push   0x804c6c4
0x8048e7c <main+730>:   push   edi
0x8048e7d <main+731>:   call   0x80485e4 <accept>
------------------------------------------------------------------------------
Breakpoint 1, main (argc=3, argv=0x1) at nweb.c:188
188           if((socketfd = accept(listenfd, (struct sockaddr *)&cli_addr, &length)) < 0)
Program received signal SIGINT, Interrupt.
0xffffe410 in ?? ()
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /home/bill/src/nweb/nweb, process 4009
$

As you can see, the information displayed by the context function is far more detailed than you normally see with the default GDB hook_stop vector. (You'll also notice that you can now access your data segment, too.) With these GDB enhancements, you can see the exact state of the CPU every time the breakpoint is reached and with every step operation you execute. Stepping through each command and watching how the register and memory values are affected is also a great way to learn the basics of the Intel machine language commands.

As is true for all programs, the code in the .gdbinit file provides endless opportunities for enhancement and improvement. By all means, this isn't the end! You're highly encouraged to use the commands described here and to add many more to the growing set of .gdbinit customizations. As you explore and use the tools, share them with the broader community so everyone grows in knowledge.

Back to top

Conclusion

More people should take up the deep study of how programming tools work, come out of their shells, and contribute to the overall communities of those who seek such knowledge. Visit some of these online communities, and even consider forming your own so that the technological innovations of the future will come more quickly into fruition.

Resources

Learn

• AIX and UNIX articles: Check out other articles written by William Zimmerly.
  
  
• GDB: The GNU Project Debugger: Get more information on the program from the home of GNU software.
  
  
• Reverse Engineering Community: Visit this site for more information on debugging processes with GDB and contributing to the community of those who take an interest in how programs work.
  
  
• "nweb: a tiny, safe Web server (static pages only)" (developerWorks, June 2004): Get the source code for the nweb Hypertext Transfer Protocol (HTTP) server.
  
  
• The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
  
  
• poweraix.org: Be sure to take advantage of poweraix.org, an information hub of the power AIX community comprising of user communities, partners, IT professionals and business users.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  
• AIX 5L Wiki: Visit this collaborative environment for technical information related to AIX.
  
  
• Podcasts: Tune in and catch up with IBM technical experts.
  
  

Get products and technologies

• IBM trial software: Build your next development project with software for download directly from developerWorks.
  
  

Discuss

• Participate in the AIX and UNIX forums:
  • AIX 5L -- technical forum
  • AIX for Developers Forum
  • Cluster Systems Management
  • IBM Support Assistant
  • Performance Tools -- technical
  • Virtualization -- technical
  • More AIX and UNIX forums
  
  

About the author

Bill Zimmerly is a knowledge engineer, a low-level systems programmer with expertise in various versions of UNIX and Microsoft® Windows®, and a free thinker who worships at the altar of Logic. Bill is also known as an unreasonable person. Unreasonable as in, "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people" (George Bernard Shaw). Creating new technologies and writing about them are his passions. He resides in rural Hillsboro, Missouri, where the air is fresh, the views are inspiring, and good wineries are all around. There's nothing quite like writing an article on UNIX shell scripting while sipping on a crystal-clear glass of Stone Hill Blush. You can contact him at bill@zimmerly.com.

Rate this article

Comments

Back to top

Table of contents

• Start simple: Clear the screen
• The importance of documentation
• Community uses for GDB
• Breakpoint aliases
• Displaying process information
• Hexadecimal and ASCII dumping commands
• Process context commands
• A debugging session with the new functionality
• Watching it live
• Conclusion
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into AIX and UNIX on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers