Obtaining a stack trace of a hung process

Answer

NOTE: Not all processes that show up in ps -ef will be able to have stack traces built on them. Old processes tend to be eventually paged out of memory and neither dbx or kdb will then be able to be used to look at the stack trace for that process.


DBX Stack Trace Instructions for building a stack trace on a hung process

In order to use dbx, the customer must first have the fileset bos.adt.debug installed.

Attach to hung process
1. Capture console output, enter:
# script

2. Find the hung process in the process table:
#  ps -ef | grep <hung process>

3. Start up dbx:
#  dbx -a <process ID>

4. To list the stack of function calls, enter:
(dbx)  where

5. Leave dbx, enter:
(dbx)  detach
(Typing quit will kill the process)

6. To leave script session and close the output file, type:
# exit

The script will be named typescript and will be located in the current
working directory.


Steps to obtain thread stack trace using the kernel debugger kdb

Using the alog process as an example.

1. Start script session to capture data:
 # script /tmp/kdb.out

2. Find the process id and convert it into hexadecimal:
    # ps -ef | grep alog
   UID     PID    PPID   C    STIME    TTY  TIME CMD
   root    1231     1    1   Jun 30      -  1:12 alog

Convert process ID 1231 to a hexadecimal number
1231 converts to 4CF

3. Start kdb
# kdb

4. Locate the process while in kdb
(0) p * | grep 4CF

Example:
pvproc+013800  78 alog   ACTIVE 4CF 004A01E  0000000002525400   0 0001

5. Find initial thread using the process slot (pslot). It is the 2nd field above
(0) p <pslot> | grep pvthread

Example:
(0) p 78 | grep pvthread


6. Locate initial thread in 'p' output
Example:
...
THREAD..... threadlist :EA005E00 <pvthread+005E00>
...

7. List the function stack for initial thread
(0) f pvthread+005E00

8. Exit from kdb
 (0) q

8. Exit out of the script session
# exit

Data will be saved in /tmp/kdb.out.


The procstack command can also be used to print the stack of a process.

# ps -ef | grep alog
    root 491752 450572   0 15:45:52  pts/4  0:00 alog

# procstack 491752
491752: alog
0xd0375da4  read(??, ??, ??) + 0x1a8
0x10001500  main(??, ??) + 0x11b0
0x10000198  __start() + 0x98


Using the pdump.sh tool

There is a script made available to customers by AIX Development Support called "pdump.sh". It will use both kdb and dbx to dump stack traces and other information about processes into a single output file. That file can be emailed to IBM Support or submitted to a PMR.

You can learn more about the pdump.sh tool and download it from here::
http://www-01.ibm.com/support/docview.wss?uid=aixtools650ae3be

Once downloaded set the permissions to be executable:
# chmod +x pdump.sh

Find the process ID of the hung process using "ps" and provide that number to the script:
# pdump.sh <PID_OF_PROCESS>

The output will be in a file in the current working directory, with the name pdump.<PID_OF_PROCESS>.<CURRENT_DATE>.out