Tools to aid debugging on the AIX platform

Debugging tips

This article discusses tools that assist AIX application developers in debugging their applications. This is especially helpful when you have to work in a customer environment where you have less debug information.

Share:

Kalyanji Chintakayala (kchintak@in.ibm.com), Developer, IBM DataStage Server Engine, IBM India

Kalyanji Chintakayala is a developer in IBM India Software Labs working on Infosphere's DataStage Server Engine. You can reach him at kchintak@in.ibm.com.



20 March 2009

Introduction

Customer reported bugs are not always easily reproducible in development environment, application crash, hang-ups and slower performance are common examples. In such cases, we need tools that can be used in a customer environment that helps in debugging the problem. A guided approach to debugging and some common problems are discussed here, along with the available tools on AIX. Note that debugging slower system performance is not discussed here.

AIX environment

The first thing we start with when a problem appears is the environment: the OS version and the hardware. This is an important step since you might want to check if you have a reproducible environment ready where you can debug.

System configuration

Run the prtconf command to see overall system configuration:

#prtconf
System Model: IBM,8204-E8A
Machine Serial Number: 06381D2
Processor Type: PowerPC_POWER6
Number Of Processors: 2
Processor Clock Speed: 4204 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 2 ibmmachine
Memory Size: 9344 MB
Good Memory Size: 9344 MB
Platform Firmware level: Not Available
Firmware Version: IBM,EL320_076
Console Login: enable
Auto Restart: true
Full Core: false

Version and maintenance levels

The following commands displays the version, release and maintenance levels of AIX:

# instfix -i|grep AIX_ML
    All filesets for 5.3.0.0_AIX_ML were found.
    All filesets for 5300-01_AIX_ML were found.
    All filesets for 5300-02_AIX_ML were found.
    All filesets for 5300-03_AIX_ML were found.
    All filesets for 5300-04_AIX_ML were found.
    All filesets for 5300-05_AIX_ML were found.
    All filesets for 5300-06_AIX_ML were found.
    All filesets for 5300-07_AIX_ML were found.

# lslpp -h bos.rte
  Fileset         Level     Action       Status       Date         Time
  ----------------------------------------------------------------------------
  Path: /usr/lib/objrepos
  bos.rte
                 5.3.0.50   COMMIT       COMPLETE     10/17/07     16:34:57
                 5.3.0.60   COMMIT       COMPLETE     03/11/08     16:08:59
                  5.3.7.0   COMMIT       COMPLETE     03/12/08     11:28:55
# oslevel -r
5300-07

CPU type and kernel type

# bootinfo -K
64
# bootinfo -y
64

Listing installed software products

# lslpp -lc|grep -i perl
/usr/lib/objrepos:perl.libext:2.1.0.10::COMMITTED:I:Perl Library Extensions :
/usr/lib/objrepos:perl.rte:5.8.2.71::COMMITTED:F:Perl Version 5 Runtime Environment:

How long system is up

#uptime
  05:16PM   up 2 days,   1:36,  4 users,  load average: 1.95, 1.90, 1.80

Tools to work on application crash

If a program is terminated, then depending on the termination type a core file could have been generated. A core file is the image of a terminated process, a dump of everything in memory at the time of the crash. A core file is generated when any of the following occurs:

  • SIGQUIT: Quit.
  • SIGILL: Invalid instruction.
  • SIGTRAP: Trace trap.
  • SIGIOT: End process.
  • SIGEMT: EMT instruction.
  • SIGFPE: Arithmetic exception, integer divide by 0, or floating-point exception.
  • SIGBUS: Specification exception.
  • SIGSEGV: Segmentation violation.
  • SIGSYS: Parameter not valid to subroutine.

Core files are not always generated when an application crashes, or they may be incomplete. If this occurs, you may need to enable core file dumps or increase core file size.

Checking core file size

#ulimit -c

This command displays the current value of core file size (called the soft limit) for the shell which is applicable for all process started from that shell. If it is zero, run the following command to increase it to its maximum value (called the hard limit):

#ulimit -c <val>

Checking hard limit for core

#ulimit -Hc

Setting core limit system wide

Edit the /etc/security/limits file and change <value> for soft and hard core size, respectively:

core = <value>
core_hard = <value>

Alternative method of setting soft limit system wide

Add the following to /etc/profile to set a soft limit:

#ulimit -S -c <value> > /dev/null 2>&1

Setting soft or hard limits for a user

chuser attribute=value username

Attributes of interest are:

  • core: size of soft limit
  • core_hard: size of hard limit
  • core_path: core file directory path enable/disable
  • core_pathname: directory to generate the core files

Changing core file setting

Use the chcore command to change the settings and lscore to view the current core settings.

Enabling fullcore dump

chdev -l sys0 -a fullcore=true

Generating core for the process that is running

The gencore utility creates a core image of each specified process. It can be then used with a debugger like dbx.

Gathering core file

The snapcore command gathers the core file, program, and libraries used by the program and compresses the information into a pax file. The file can then be transmitted to a debug environment and can be used to identify and resolve a problem with the application.

snapcore -r<core file name> <program name>

The pax file is created in the /tmp/snapcore directory.

How to determine where the core file is created and which program caused it

If a core file has been created, there should be an error log entry created by the error logging process. This usually starts when the first software failure occurs.

  1. Retrieve the error log:
    # errpt -a
    LABEL:          CORE_DUMP
    IDENTIFIER:     C69F5C9B
    
    Date/Time:       Fri Nov 13 17:04:55 IST 2009
    Sequence Number: 235168
    Machine Id:      000381D2D900
    Node Id:         ibmmachine
    Class:           S
    Type:            PERM
    Resource Name:   SYSPROC
    
    Description
    SOFTWARE PROGRAM ABNORMALLY TERMINATED
    
    Probable Causes
    SOFTWARE PROGRAM
    
    User Causes
    USER GENERATED SIGNAL
    
            Recommended Actions
            CORRECT THEN RETRY
    
    Failure Causes
    SOFTWARE PROGRAM
    
            Recommended Actions
            RERUN THE APPLICATION PROGRAM
            IF PROBLEM PERSISTS THEN DO THE FOLLOWING
            CONTACT APPROPRIATE SERVICE REPRESENTATIVE
    
    Detail Data
    SIGNAL NUMBER
              11
    USER'S PROCESS ID:
                    765972
    FILE SYSTEM SERIAL NUMBER
               8
    INODE NUMBER
          352516
    CORE FILE NAME
    /opt/IBM/InformationServer/Server/Projects/sample1/core
    PROGRAM NAME
    dsapi_slave

    The program which generated the core is mentioned under PROGRAM_NAME.

  2. Displaying errors with reference to time

    To display a detailed report of all errors logged in the past 24 hours, use the errpt command as follows:

    # date
    Fri Nov 13 18:18:33 IST 2009
    # errpt -a -s 1112181809

Which application created the core?

#lquerypv -h core 6b0 64

The executable is located between the pipes on the right hand side of the output.

000006B0 7FFFFFFF FFFFFFFF 7FFFFFFF FFFFFFFF |................|
000006C0 00000000 000007D0 7FFFFFFF FFFFFFFF |................|
000006D0 00120000 1312C9C0 00000000 00000017 |................|
000006E0 6E657473 63617065 5F616978 34000000 |netscape_aix4...|
000006F0 00000000 00000000 00000000 00000000 |................|
00000700 00000000 00000000 00000000 00000ADB |................|
00000710 00000000 000008BF 00000000 00000A1E |................|

Examining core file

Run dbx on the binary executable that caused the core dump. This will display the offending call:

#dbx exe core

System settings useful for debugging

Listing sys0:

lsattr -El sys0

Useful attributes:

  • autorestart: Automatically REBOOT system after a crash
  • fullcore: Enable/disable full CORE dump
  • maxuproc: Maximum number of PROCESSES allowed per user

Changing the system attributes

chdev -l sys0 -a attribute=value

Process inspection tools

There are a bunch of tools on AIX for inspecting processes for application errors, hangs, and crashes. We will discuss some of them here.

The following tools can be used to inspect the process or core in question. All the commands start with proc<cmd>. Special care should be taken while inspecting process in production environment since these tools actually stop (not kill) the process while they inspect.

  • procstack: prints stack trace of the process
  • procflags: prints pending and held signals for the process
  • procsig: prints signal actions and handlers for the process
  • procfiles: Report fstat and fcntl information for all open files in each process
  • procwdx: prints current working directory of the process procstop, procrun: to stop (not kill) and to run the stopped process
  • proctree: Print the process trees containing the specified pids or users, with child processes indented from their respective parent processes.

Watching live a process

truss produces a trace of the system calls it performs, the signals it receives, and the machine faults it incurs. By default, user level functions are not traced. To enable tracing for all user level functions do the following:

truss  -u '*'  -p <pid>

Useful options:

  • -p: Process id
  • -u [!] [LibraryName [...]::[!]FunctionName [...] ]: Traces dynamically loaded user level function calls from user libraries.
  • -a: Shows the argument strings that are passed in each exec() system call.
  • -f: Follows all children created by fork() or vfork() and includes their signals, faults, and system calls in the trace output.
  • -m [!]Fault: Traces the listed(see the sys/procfs.h header file) machine faults in the process.
  • -s [!] Signal: Permits listing Signals to trace or exclude.

truss'ing an SUID process

If you want to truss a command which runs as another user under SUID, you will not be allowed to do so as the system identifies it as not belonging to your user. The following error displays:

# truss -deaf -o truss.out program

truss: 0915-015 Cannot create subject process.
wait4all: i: 0, status: 32512, pid: 643282, created: 0

To truss such commands, do the following:

  • Login as the user who you need to investigate and find the PID of your shell using the ps command.
  • Start a new session as root and truss the shell session.
  • This new session will now log all the activity in the original shell. Run the failing command and then stop the truss. The truss.out file can be investigated to find the failure.

Knowing names of the files opened by a process

In a typical database system environment, or applications that use extensive usage of file handling, it might be important to know names of files owned by a process for debugging the problem.

  1. List the names of the files owned by process:

    procfiles -n <pid>
  2. If you know inode number then:
    • ncheck generates path names from i-node numbers
      	ncheck - i <inode>
    • List the files and grep for the inode
      	ls -ail |grep <inode>

Process hangs while connecting or accepting TCP connections

netstat -a |grep <process name>

If client process status field is in FIN_WAIT state for long periods of time or the server process status field is in CLOSE_WAIT state for long periods of time, the process are said to be hanging or a deadlock could have occurred.

Socket to Process ID mapping

Run netstat -Aan, where -A shows the address of any protocol control blocks associated with the sockets.

#netstat -Ana|grep 31538
f10006000041c398 tcp4       0      0  *.31538            *.*                LISTEN
f10006000677d398 tcp4       0      0  9.122.87.107.31538 9.122.87.51.2500   ESTABLISHED
f100060006affb98 tcp4       0      0  9.122.87.107.31538 9.122.87.51.2511   ESTABLISHED
f1000600066d1398 tcp4       0      0  9.122.87.107.31538 9.122.87.51.2521   ESTABLISHED

Run kdb and issue sockinfo on the address for the socket in question.

(0)> sockinfo f10006000677d398 tcpcb
---- TCPCB ----(@ F10006000677D398)----
    seg_next......@F10006000677D398  seg_prev......@F10006000677D398
    t_softerror... 00000000 t_state....... 00000004 (ESTABLISHED)
    t_timer....... 00000000 (TCPT_REXMT)
....
proc/fd: fd: 4
              SLOT NAME     STATE      PID    PPID          ADSPACE  CL #THS

pvproc+01B000  108*dsapi_sl ACTIVE 006C0D0 00B206C 000000002E707590   0 0001

Where PID is represented in hexadecimal.

Check for hangs from CPU usage

#ps -fp <pid>

Check the TIME field. If it is constant over time then a probable deadlock or hang could have occurred.

#ps -mp <pid> -o THREAD

Monitors threads activity in a process.


Tools to work on process memory

Data segments settings

LDR_CNTRL environment variable controls the number of data segments that a process can use. The following example defines one additional data segment:

export LDR_CNTRL=MAXDATA=0x10000000
start the process
unset LDR_CNTRL

This value greatly effects some of the memory related issues on AIX. MAXDATA controls the amount of malloc'd memory and MAXDATA is changed using LDR_CNTRL=MAXDATA=0xN0000000 (N = # of segments).

On 32-bit systems, the default address space model is that it uses a single segment for both user data and stack with maximum aggregate size close to 256MB. If your applications requires more than that then "Large or Very large address-space model" can be used by setting MAXDATA.

See AIX documents for more on Large program support.

ldedit command can also be used to change the maxdata settings in the executable itself:

ldedit -bmaxdata:0x80000000 sampleexec

For 32-bit programs under Large address-space model the maximum value allowed is 0x80000000 and under Very Large address-space model it is 0xD0000000.

For 64-bit programs any value can be specified, but the data area cannot extend past 0x06FFFFFFFFFFFFF8.

Memory usage of a process

The ps command reports malloc'd memory and does not include mmap'd memory. svmon reports complete process memory utilization.

#svmon -P <pid> -m -r -i <interval>

Late and early allocation

Memory and paging space allocation by default is late. The PSALLOC environment variable controls the mechanism of allocation.

#export PSALLOC=early

By default(late), when malloc is called, no paging space is assigned until it is referenced. It is possible for malloc to overcommit (not enough backing storage) and some other process may get the resource before the current process, resulting in a failure. Setting PSALLOC to early guarantees as much paging space as requested by the memory allocation request.

Shared memory settings

To print information about active shared memory segments

#ipcs -mop

To remove shared memory segments

ipcrm [ -m SharedMemoryID ] [ -M SharedMemoryKey ]

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=503375
ArticleTitle=Tools to aid debugging on the AIX platform
publish-date=03202009