The previous article on this topic, "Fun with strace and GDB," covered the basics of using these tools to explore your system and attach to programs that are already running to see what they were doing. This article presses on to customize the debugger to make the experience more personal and efficient.
When GDB, the GNU Project Debugger, starts up, it looks for a file in the current user's home directory called .gdbinit; if the file exists, GDB executes all the commands in the file. Normally, this file is used for simple configuration commands, such as setting the default assembler format desired (Intel® or Motorola) or default radix to display input and output data (decimal or hexadecimal). It can also read a macro-coding language that allows more powerful customizations. The language follows this basic format:
define <command> <code> end document <command> <help text> end
The command is known as a user command. All other standard GDB commands can be combined with flow-control instructions and passed parameters to create a language that allows you to customize the behavior of the debugger for the specific application being debugged.
It's always good to start simple and build up from there. Start the xterm, fire
up your favorite editor, and let's begin creating a useful .gdbinit file! The output from
debuggers can be messy and, as a matter of personal preference, many people like to be able to clear
the screen when working with any tool that has the potential for creating cluttering. GDB has
no built-in command for clearing the screen, but it can call shell functions; the
following code steps outside the debugger to clear the xterm console with the
define cls shell clear end document cls Clears the screen with a simple command. end
The upper portion of the definition, bounded by the
define ... end verbs,
frames the code that is executed when the command is invoked.
The lower portion of the definition, bounded by
document ... end, is
used by the GDB command interpreter to display help text associated with the
when you type
After you type the code into the .gdbinit file, fire up GDB and enter the
Your screen clears, and all you see is the GDB prompt. Your journey into GDB customization
If you enter the
help user command, you see a summary of all the user commands you've entered in the .gdbinit file. The designers of the .gdbinit user-defined commands have provided an important feature that you shouldn't ignore when you're writing your own commands: the
document ... end clause. Maintaining functional documentation of how commands work becomes critical as the number of these commands increase.
You might have already experienced the problem. Suppose you wrote some code from a few years ago; then, when you revisit it, perhaps to fix a bug or modify it by adding a new feature, you realize that you're having a hard time understanding your own code. Good programmers make it a habit to keep code short, simple, and well-documented so that it's maintainable.
What's true for programming code in general is true for debugger code. Keeping careful notes and well-documented code will serve you well as you work your way through this most rewarding of careers.
Human beings learn new things many ways, including by studying what others have done. Budding automotive engineers start by opening the hood of their first car, pulling out their tools, and removing parts to clean and study. Such an exercise lets them keep a clean machine while learning how the automobile engine works.
Budding computer scientists are no different in that they want to see just how programs work -- how they interact with dynamic libraries and the native operating system. The tool for seeing how these things work is the debugger. Computer programming is a complex activity, and by being able to interact with a community of like-minded people, asking questions and getting answers, new computer scientists can satisfy their need for knowledge.
In the global programming community, there are always a healthy number of people who hunger and thirst for knowledge. They aren't content with just running programs on their computers -- they want to know more. They want to know how those programs run, and they enjoy exploring the functionality of their systems with the best tool available for the purpose: the debugger. By reverse engineering, a way of learning how programs work by running them under a debugger and keeping careful notes of how they do what they do, you can learn a lot from what the authors of the program being studied have done. Many low-level details involved in programming are undocumented; the only way to learn about them is to see them in action.
Reverse engineering has an undeserved reputation as a black art practiced only by hackers and criminals intent on breaking copy-protection systems and writing worms and viruses to inflict harm on the world of computers. Although there are such people, the vast majority of those who study how programs work, using debuggers and reverse engineering, are the present and future software engineers who want and need to know how these things work. They have formed online communities to share their knowledge and discoveries; discouraging this activity is destructive and retards the future progress of computer science.
Many of the user functions defined in this article come from such communities of hungry knowledge seekers. If you want to know more about them, you're encouraged to study the Web sites referenced in the Resources section of this document.
It's a well-known fact that many of the GDB commands are too verbose. Even
though they can be abbreviated, the GDB macro language allows for even greater
simplifications. Commands, such as
info breakpoints, can
be made as simple as
bpl. Listing 1 shows
a great set of such simple and highly useful breakpoint alias user commands
to edit into your growing .gdbinit file.
Listing 1: Breakpoint alias commands
define bpl info breakpoints end document bpl List breakpoints end define bp set $SHOW_CONTEXT = 1 break * $arg0 end document bp Set a breakpoint on address Usage: bp addr end define bpc clear $arg0 end document bpc Clear breakpoint at function/address Usage: bpc addr end define bpe enable $arg0 end document bpe Enable breakpoint # Usage: bpe num end define bpd disable $arg0 end document bpd Disable breakpoint # Usage: bpd num end define bpt set $SHOW_CONTEXT = 1 tbreak $arg0 end document bpt Set a temporary breakpoint on address Usage: bpt addr end define bpm set $SHOW_CONTEXT = 1 awatch $arg0 end document bpm Set a read/write breakpoint on address Usage: bpm addr end
Once you get used to using breakpoint alias commands, your debugging sessions become more rewarding; these commands greatly increase the debugger's effectiveness, because you can accomplish more with less effort.
The GDB user-defined commands can be called by other user-defined commands, leading to greater effectiveness for them all. This is the incremental nature of programming languages -- writing lower-level functions that are called by progressively higher-level functions until the tools conveniently do what you want them to do with minimal effort on your part. The next set of GDB definitions to incorporate into your .gdbinit file display useful process information when they're invoked, as shown in Listing 2.
Listing 2: Process information commands
define argv show args end document argv Print program arguments end define stack info stack end document stack Print call stack end define frame info frame info args info locals end document frame Print stack frame end define flags if (($eflags >> 0xB) & 1 ) printf "O " else printf "o " end if (($eflags >> 0xA) & 1 ) printf "D " else printf "d " end if (($eflags >> 9) & 1 ) printf "I " else printf "i " end if (($eflags >> 8) & 1 ) printf "T " else printf "t " end if (($eflags >> 7) & 1 ) printf "S " else printf "s " end if (($eflags >> 6) & 1 ) printf "Z " else printf "z " end if (($eflags >> 4) & 1 ) printf "A " else printf "a " end if (($eflags >> 2) & 1 ) printf "P " else printf "p " end if ($eflags & 1) printf "C " else printf "c " end printf "\n" end document flags Print flags register end define eflags printf " OF <%d> DF <%d> IF <%d> TF <%d>",\ (($eflags >> 0xB) & 1 ), (($eflags >> 0xA) & 1 ), \ (($eflags >> 9) & 1 ), (($eflags >> 8) & 1 ) printf " SF <%d> ZF <%d> AF <%d> PF <%d> CF <%d>\n",\ (($eflags >> 7) & 1 ), (($eflags >> 6) & 1 ),\ (($eflags >> 4) & 1 ), (($eflags >> 2) & 1 ), ($eflags & 1) printf " ID <%d> VIP <%d> VIF <%d> AC <%d>",\ (($eflags >> 0x15) & 1 ), (($eflags >> 0x14) & 1 ), \ (($eflags >> 0x13) & 1 ), (($eflags >> 0x12) & 1 ) printf " VM <%d> RF <%d> NT <%d> IOPL <%d>\n",\ (($eflags >> 0x11) & 1 ), (($eflags >> 0x10) & 1 ),\ (($eflags >> 0xE) & 1 ), (($eflags >> 0xC) & 3 ) end document eflags Print entire eflags register end define reg printf " eax:%08X ebx:%08X ecx:%08X ", $eax, $ebx, $ecx printf " edx:%08X eflags:%08X\n", $edx, $eflags printf " esi:%08X edi:%08X esp:%08X ", $esi, $edi, $esp printf " ebp:%08X eip:%08X\n", $ebp, $eip printf " cs:%04X ds:%04X es:%04X", $cs, $ds, $es printf " fs:%04X gs:%04X ss:%04X ", $fs, $gs, $ss flags end document reg Print CPU registers end define func info functions end document func Print functions in target end define var info variables end document var Print variables (symbols) in target end define lib info sharedlibrary end document lib Print shared libraries linked to target end define sig info signals end document sig Print signal actions for target end define thread info threads end document thread Print threads in target end define u info udot end document u Print kernel 'user' struct for target end define dis disassemble $arg0 end document dis Disassemble address Usage: dis addr end
The next set of definitions to incorporate into the .gdbinit file includes enhanced hexadecimal and ASCII dump functions, as shown in Listing 3. Programmers take note: If you want to create great software, then add the ability to program macros, allowing your community of users to be able to enhance your tool to suit their own tastes. GDB is great software!
Listing 3: Hexadecimal and ASCII dumping commands
define ascii_char set $_c=*(unsigned char *)($arg0) if ( $_c < 0x20 || $_c > 0x7E ) printf "." else printf "%c", $_c end end document ascii_char Print the ASCII value of arg0 or '.' if value is unprintable end define hex_quad printf "%02X %02X %02X %02X %02X %02X %02X %02X", \ *(unsigned char*)($arg0), *(unsigned char*)($arg0 + 1), \ *(unsigned char*)($arg0 + 2), *(unsigned char*)($arg0 + 3), \ *(unsigned char*)($arg0 + 4), *(unsigned char*)($arg0 + 5), \ *(unsigned char*)($arg0 + 6), *(unsigned char*)($arg0 + 7) end document hex_quad Print eight hexadecimal bytes starting at arg0 end define hexdump printf "%08X : ", $arg0 hex_quad $arg0 printf " - " hex_quad ($arg0+8) printf " " ascii_char ($arg0) ascii_char ($arg0+1) ascii_char ($arg0+2) ascii_char ($arg0+3) ascii_char ($arg0+4) ascii_char ($arg0+5) ascii_char ($arg0+6) ascii_char ($arg0+7) ascii_char ($arg0+8) ascii_char ($arg0+9) ascii_char ($arg0+0xA) ascii_char ($arg0+0xB) ascii_char ($arg0+0xC) ascii_char ($arg0+0xD) ascii_char ($arg0+0xE) ascii_char ($arg0+0xF) printf "\n" end document hexdump Display a 16-byte hex/ASCII dump of arg0 end define ddump printf "[%04X:%08X]------------------------", $ds, $data_addr printf "---------------------------------[ data]\n" set $_count=0 while ( $_count < $arg0 ) set $_i=($_count*0x10) hexdump ($data_addr+$_i) set $_count++ end end document ddump Display $arg0 lines of hexdump for address $data_addr end define dd if ( ($arg0 & 0x40000000) || ($arg0 & 0x08000000) || ($arg0 & 0xBF000000) ) set $data_addr=$arg0 ddump 0x10 else printf "Invalid address: %08X\n", $arg0 end end document dd Display 16 lines of a hex dump for $arg0 end define datawin if ( ($esi & 0x40000000) || ($esi & 0x08000000) || ($esi & 0xBF000000) ) set $data_addr=$esi else if ( ($edi & 0x40000000) || ($edi & 0x08000000) || ($edi & 0xBF000000) ) set $data_addr=$edi else if ( ($eax & 0x40000000) || ($eax & 0x08000000) || \ ($eax & 0xBF000000) ) set $data_addr=$eax else set $data_addr=$esp end end end ddump 2 end document datawin Display esi, edi, eax, or esp in the data window end
Finally, when you're debugging a running process, it's often necessary to get an overall view of the process context. The useful process context commands in Listing 4 are built by using the previously defined data dumping functions.
Listing 4: Process context commands
define context printf "_______________________________________" printf "________________________________________\n" reg printf "[%04X:%08X]------------------------", $ss, $esp printf "---------------------------------[stack]\n" hexdump $sp+0x30 hexdump $sp+0x20 hexdump $sp+0x10 hexdump $sp datawin printf "[%04X:%08X]------------------------", $cs, $eip printf "---------------------------------[ code]\n" x /6i $pc printf "---------------------------------------" printf "---------------------------------------\n" end document context Print regs, stack, ds:esi, and disassemble cs:eip end define context-on set $SHOW_CONTEXT = 1 end document context-on Enable display of context on every program stop end define context-off set $SHOW_CONTEXT = 1 end document context-on Disable display of context on every program stop end # Calls "context" at every breakpoint. define hook-stop context end # Init parameters set output-radix 0x10 set input-radix 0x10 set disassembly-flavor intel
hook-stop is a special definition that GDB calls at every breakpoint
event. In this case, the
context listing is generated so that you can clearly see
the effect of each instruction being executed by the processor.
Let's experiment with your new set of tools to see how they work when debugging an old friend, the nweb server code by IBM developerWorks contributing writer Nigel Griffiths. (See the Resources section for a link to Nigel's article "nweb: a tiny, safe Web server (static pages only).")
After downloading the es-nweb.zip file to your $HOME/downloads directory, type the following commands to extract, compile, and launch nweb. (Please note that this assumes you're compiling the program to a Linux® workstation with the Intel Pentium as the central processing unit (CPU) -- the .gdbinit code was written for the Intel Pentium-type processor and compatibles only.)
$ cd src $ mkdir nweb $ cd nweb $ unzip $HOME/downloads/es-nweb.zip $ gcc -ggdb -O -DLINUX nweb.c -o nweb $ ./nweb 9090 $HOME/src/nweb &
-ggdb option in this example differs from Nigel's
article in that it tells the GNU Compiler Collection (GCC) to optimize the program for debugging with GDB.
Next, to verify that the nweb server is running, use the
to check it:
$ ps PID TTY TIME CMD 2913 pts/5 00:00:00 bash 4009 pts/5 00:00:00 nweb 4011 pts/5 00:00:00 ps
Finally, launch a Web browser on your computer and type http://localhost:9090 in the address bar.
Next, start GDB and, as before, attach to the currently running nweb instance, as shown in Listing 5.
Listing 5: Launch GDB
$ gdb --quiet (gdb) attach 4009 Attaching to process 4009 Reading symbols from /home/bill/src/nweb/nweb...done. Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 _______________________________________________________________________________ eax:FFFFFE00 ebx:00000005 ecx:BFFFF680 edx:00000001 eflags:00000246 esi:00000005 edi:00000000 esp:BFFFF66C ebp:BFFFF6A8 eip:FFFFE410 cs:0073 ds:007B es:007B fs:0000 gs:0033 ss:007B o d I t s Z a P c [007B:BFFFF66C]---------------------------------------------------------[stack] BFFFF69C : 14 0A 13 42 60 53 01 40 - 24 8F 04 08 C8 F6 FF BF ...B`S.@$....... BFFFF68C : A6 8E 04 08 14 0A 13 42 - 70 C6 00 40 10 00 00 00 .......Bp..@.... BFFFF67C : 82 8E 04 08 00 00 00 00 - C4 C6 04 08 98 F6 FF BF ................ BFFFF66C : A8 F6 FF BF 01 00 00 00 - 80 F6 FF BF 81 EA 0D 42 ...............B [007B:FFFFFE00]---------------------------------------------------------[ data] FFFFFE00 : Error while running hook_stop: Cannot access memory at address 0xfffffe00 0xffffe410 in ?? () (gdb)
-quiet option tells the GDB Debugger to display only its prompt
and not all the other startup information that it typically displays. If you want the extra text, leave the
attach 4009 command starts debugging the currently running
nweb server, and the GDB Debugger responds in kind by reading all the symbolic information about
the process it can.
You'll notice that the
context code ran and displayed a lot of useful information about
the current process, but it was unable to access memory in the data segment. This is not a serious
problem, and it should be ignored. Sometimes, the protection scheme of a protected mode processor
won't let you see all that you might want to see. In this case, it isn't important.
Next, use the
info command to list information
about the program you're exploring (see Listing 6).
Listing 6: The info command lists program information
(gdb) info proc process 4009 cmdline = './nweb' cwd = '/home/bill/src/nweb' exe = '/home/bill/src/nweb/nweb' (gdb)
Because you're watching an actual running program, you can set breakpoints and then see the program as it replies to browser requests and transmits .html and .jpg files to the browser making the request. Listing 7 shows how you can do so.
Listing 7: Set breakpoints
(gdb) b 188 Breakpoint 1 at 0x8048e70: file nweb.c, line 188. (gdb) commands 1 Type commands for when breakpoint 1 is hit, one per line. End with a line saying just "end". >continue >end (gdb) c Continuing.
At this point, the GDB Debugger is set to break at the line where the nweb server accepts
browser requests; the debugger will simply display the request and continue processing other
requests without interrupting the running program. Refresh the
page in your browser a few times, and watch the GDB Debugger display the breakpoint and
While refreshing the browser page, you should see breakpoint information, such as that shown
in Listing 8, scrolling in the GDB Debugger xterm. You can stop debugging in the nweb server
by pressing Ctrl+C. After stopping the tracing, you can exit the GDB Debugger
by typing the
Listing 8: Breakpoint information in the GDB Debugger xterm
_______________________________________________________________________________ eax:00000000 ebx:00000001 ecx:00000000 edx:00000001 eflags:00000206 esi:00000006 edi:00000000 esp:BFFFF690 ebp:BFFFF6A8 eip:08048E70 cs:0073 ds:007B es:007B fs:0000 gs:0033 ss:007B o d I t s z a P c [007B:BFFFF690]---------------------------------------------------------[stack] BFFFF6C0 : 03 00 00 00 D4 86 04 08 - 00 00 00 00 F5 86 04 08 ................ BFFFF6B0 : 03 00 00 00 F4 F6 FF BF - 04 F7 FF BF 2C 58 01 40 ............,X.@ BFFFF6A0 : 60 53 01 40 24 8F 04 08 - C8 F6 FF BF 04 55 01 42 `S.@$........U.B BFFFF690 : 14 0A 13 42 70 C6 00 40 - 10 00 00 00 14 0A 13 42 ...Bp..@.......B [007B:BFFFF690]---------------------------------------------------------[ data] BFFFF690 : 14 0A 13 42 70 C6 00 40 - 10 00 00 00 14 0A 13 42 ...Bp..@.......B BFFFF6A0 : 60 53 01 40 24 8F 04 08 - C8 F6 FF BF 04 55 01 42 `S.@$........U.B [0073:08048E70]---------------------------------------------------------[ code] 0x8048e70 <main+718>: sub esp,0x4 0x8048e73 <main+721>: lea eax,[ebp-16] 0x8048e76 <main+724>: push eax 0x8048e77 <main+725>: push 0x804c6c4 0x8048e7c <main+730>: push edi 0x8048e7d <main+731>: call 0x80485e4 <accept> ------------------------------------------------------------------------------ Breakpoint 1, main (argc=3, argv=0x1) at nweb.c:188 188 if((socketfd = accept(listenfd, (struct sockaddr *)&cli_addr, &length)) < 0) Program received signal SIGINT, Interrupt. 0xffffe410 in ?? () (gdb) quit The program is running. Quit anyway (and detach it)? (y or n) y Detaching from program: /home/bill/src/nweb/nweb, process 4009 $
As you can see, the information displayed by the
context function is far more
detailed than you normally see with the default GDB
hook_stop vector. (You'll also notice that you can now access your data segment, too.) With these
GDB enhancements, you can see the exact state of the CPU every time
the breakpoint is reached and with every step operation you execute.
Stepping through each command and watching how the register and memory values
are affected is also a great way to learn the basics of the Intel machine language commands.
As is true for all programs, the code in the .gdbinit file provides endless opportunities for enhancement and improvement. By all means, this isn't the end! You're highly encouraged to use the commands described here and to add many more to the growing set of .gdbinit customizations. As you explore and use the tools, share them with the broader community so everyone grows in knowledge.
More people should take up the deep study of how programming tools work, come out of their shells, and contribute to the overall communities of those who seek such knowledge. Visit some of these online communities, and even consider forming your own so that the technological innovations of the future will come more quickly into fruition.
- AIX and UNIX articles: Check out other articles written by William Zimmerly.
GDB: The GNU Project Debugger: Get more information on the program from the home of GNU software.
- Reverse Engineering Community: Visit this site for more information on debugging processes with GDB and contributing to the community of those who
take an interest in how programs work.
"nweb: a tiny, safe
Web server (static pages only)" (developerWorks, June 2004): Get the source code for the nweb Hypertext Transfer Protocol
The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
- poweraix.org: Be sure to take advantage of poweraix.org, an information hub of the power AIX community comprising of user communities, partners, IT professionals and business users.
developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
AIX 5L Wiki: Visit this collaborative environment for technical information related to AIX.
Podcasts: Tune in and catch up with IBM technical experts.
Get products and technologies
- IBM trial software: Build your next development project with software for download directly from developerWorks.
Participate in the AIX and UNIX forums:
- AIX 5L -- technical forum
- AIX for Developers Forum
- Cluster Systems Management
- IBM Support Assistant
- Performance Tools -- technical
- Virtualization -- technical
- More AIX and UNIX forums
Bill Zimmerly is a knowledge engineer, a low-level systems programmer with expertise in various versions of UNIX and Microsoft® Windows®, and a free thinker who worships at the altar of Logic. Bill is also known as an unreasonable person. Unreasonable as in, "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people" (George Bernard Shaw). Creating new technologies and writing about them are his passions. He resides in rural Hillsboro, Missouri, where the air is fresh, the views are inspiring, and good wineries are all around. There's nothing quite like writing an article on UNIX shell scripting while sipping on a crystal-clear glass of Stone Hill Blush. You can contact him at firstname.lastname@example.org.