Answer

Program execution errors on the system can be caused by software installation problems such as failed updates, missing or broken filesets or commands, or filesystem damage.

Execution errors can also cause a hang on reboot, if the program is necessary for the system to function.

If you believe a recent software update has been done improperly, or did not completely install all required filesets, you can use the /usr/bin/lppchk utility and other commands to test it. 


Examples of running lppchk

If you run most lppchk commands and the system is clean, no trouble found, they will come back to the prompt with no output.


# lppchk -c bos.rte.security
#


So to make sure there is SOME output we suggest using verbose mode:
 
# lppchk -cm3 bos.rte.security
lppchk: 0504-230  251 files have been checked.
lppchk: 0504-230  183 files have been checked.
#


Typical lppchk and trustchk options we suggest are:


# lppchk -v
# lppchk -cm3
# lppchk -fm3
# lppchk -l
# trustchk -n ALL


Below are some common execution errors and the possible solutions.

Problem 1: Bad library or file.
The error seen by the user is:

$ ps -ef
Could not load program
/usr/bin/ps:
Could not load module /usr/lib/libcorcfg.a(shr.o).
       Dependent module libsrc.a(shr.o) could not be loaded.
       File libsrc.a is not an
         archive or the file could not be read properly.
System error: Exec format error

Notice a few things here:
1. The program we are trying to execute is /usr/bin/ps
2. It is trying to load a necessary library /usr/lib/libcorcfg.a and access an object inside of that library (shr.o).
3. It fails because it cannot load that shr.o object
4. The failure is that the archive could not be read properly.

This means the library /usr/lib/libcorcfg.a is corrupt.


Solution:

1. Check the size of the library to see if it has been accidentally zeroed out.  If so restore it from a backup.
2. Check errpt for filesystem corruption errors against the /usr filesystem.
3. Boot into maintenance mode and run fsck on /usr if you believe it is corrupt.
4. Restore the damaged library from a backup or extract it from the fileset it comes in.
5. If fsck cannot repair the damaged filesystem, or if there are mulitple damaged filesystems, check errpt for disk or other errors that may be the root cause of these problems.  Correct those problems and restore the OS if necessary.

Problem 2: Missing link target
$ xterm
Could not load program xterm:
Symbol resolution failed for /usr/lib/libXext.a[shr_64.o] because:
       Symbol _XEatDataWords (number 27) is not exported from dependent
         module /usr/lib/libX11.a[shr_64.o].
System error: Error 0
Examine .loader section symbols with the 'dump -Tv' command.

In the above case the problem was a broken link, told to us by lppchk:

lppchk:  Existing entry at /usr/lib/libX11.a is not a link to
/usr/lpp/X11/lib/R7/libX11.a.

Solution:
Recreate the link to the library or restore the library if it is missing

$ ls -l /usr/lib/libX11.a
lrwxrwxrwx   1 bin      bin              28 Apr 15 2008  /usr/lib/libX11.a -> /usr/lpp/X11/lib/R6/libX11.a


If the link is there, does the target library exist?


# ls -lH /usr/lib/libX11.a
ls: 0653-341 The file /usr/lpp/X11/lib/R6/libX11.a does not exist.


In this case the target of the link no longer exists on the system. Restore the library from a backup.


Problem 3: AIX has been updated but not rebooted

Could not load program /usr/bin/ksh:
Symbol resolution failed for /usr/lib/libc.a(shr.o) because:
        Symbol fobject_stat (number 71) is not exported from dependent module /unix.
        Symbol vpm_get_stats (number 278) is not exported from dependent module /unix.
        Symbol vpmctl (number 279) is not exported from dependent module /unix.
        Symbol rslb_test_ctl (number 423) is not exported from dependent module /unix.
        Symbol _etrc_get_probes (number 437) is not exported from dependent module /unix.
        Symbol _etrc_getsession_info (number 438) is not exported from dependent module /unix.
        Symbol accel_compress (number 540) is not exported from dependent module /unix.
        Symbol accel_decompress (number 541) is not exported from dependent module /unix.
System error: Error 0
Examine .loader section symbols with the 'dump -Tv' command.


Here we can see the system is trying to execute the ksh shell, and finding that some necessary symbols the C library "libc.a" it needs are not present in the /unix kernel that is running.


Typically this will mean that the system has been upgraded, but not rebooted on the new kernel.  The kernel that is running in memory must match the C library, other libraries, and commands on the system.


Solution:
Possibly run bosboot and reboot.

First check the date of last reboot:

$ uptime
  10:54AM   up 100 days, 42 mins,  1 user,  load average: 1.53, 1.65, 1.71
$ who -r
   .        run-level 2 Mar 23 10:12       2    0    S


So in our example the system was rebooted on March 23


Now check when the kernel, libc and command fileset (in this case ksh) were updated:


$ lslpp -h bos.mp64
  Fileset         Level     Action       Status       Date         Time
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.mp64
                 7.1.3.45   COMMIT       COMPLETE     06/07/16     15:58:53

Path: /etc/objrepos
  bos.mp64
                 7.1.3.45   COMMIT       COMPLETE     06/07/16     15:58:55

$ lslpp -h bos.rte.libc
  Fileset         Level     Action       Status       Date         Time
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.rte.libc
                 7.1.3.45   COMMIT       COMPLETE     06/07/16     15:41:15

Path: /etc/objrepos
  bos.rte.libc
                 7.1.3.45   COMMIT       COMPLETE     06/07/16     15:41:15

$ lslpp -h bos.rte.shell
  Fileset         Level     Action       Status       Date         Time
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.rte.shell
                 7.1.3.45   COMMIT       COMPLETE     06/07/16     15:41:15

Path: /etc/objrepos
  bos.rte.shell
                 7.1.3.45   COMMIT       COMPLETE     06/07/16     15:41:15


The filesets were updated on June 7 so they have not rebooted on the new kernel and libraries yet. Have the customer run bosboot and schedule a reboot soon.



Problem 4: AIX update or installation is not correct
# bootlist -m normal -o
Could not load program bootlist:
Symbol resolution failed for bootlist_chrp because:
       Symbol ___strcmp (number 8) is not exported from dependent
         module /usr/lib/libc.a(shr.o).
       Symbol ___strcpy (number 13) is not exported from dependent
         module /usr/lib/libc.a(shr.o).
Examine .loader section symbols with the 'dump -Tv' command

Solution:
Run oslevel -s to determine if the system is at the correct level. If it shows a lower level than it should, they are probably missing pre-requisite filesets.

Use lppchk and trustchk as in the top of this technote. If any filesets are missing or partially installed, install them and retry the command.


Problem 5: IFIX installed is not correct

If an interim fix (aka ifix) is installed and replaces one of the components necessary for the command, it may need to be rebuilt. For example an updated kernel may not be correctly linked with the right level of library and cause errors such as:

$ vmstat

Could not load program vmstat:
Symbol resolution failed for vmstat because:
        Symbol ptx_get_numdkstats (number 77) is not exported from dependent module /unix.
        Symbol ptx_get_dkstats (number 78) is not exported from dependent module /unix.

Solution:
List out which libraries and other object files (such as the Unix kernel) are used by the command. Using our above example of vmstat:

$ which vmstat
/usr/bin/vmstat

$ ldd /usr/bin/vmstat
/usr/bin/vmstat needs:
         /usr/lib/libc.a(shr_64.o)
         /usr/lib/libwlm.a(shr_64.o)
         /usr/lib/libperfstat.a(shr_64.o)
         /usr/lib/libcorcfg.a(shr_64.o)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)
         /usr/lib/libpthreads.a(shr_xpg5_64.o)
         /usr/lib/libcfg.a(shr_64.o)
         /usr/lib/libodm.a(shr_64.o)
         /usr/lib/liblvm.a(shr_64.o)
         /usr/lib/libsrc.a(shr_64.o)

Check the installed ifixes using:

$ emgr -l

And if any of them supply a necessary component listed above, the ifix will need to be rebuilt. If the ifix is old enough it may already be in a new Service Pack. If that is the case update to the latest Service Pack for your Technology Level.



ADDITIONAL NOTE FOR AIX 7.2 ONLY

There is an APAR for AIX 7.2 that can cause this type of error:


#  mirrorvg rootvg
0516-1804 chvg: The quorum change takes effect immediately.
exec(): 0509-036 Cannot load program lresynclv because of the following
errors:
     0509-022 Cannot load module /usr/lib/libhmc.a(shr.o).
     0509-150   Dependent module /opt/pware/lib/libxml2.a(libxml2.
shr.o) could not be loaded.
     0509-152   Member libxml2.shr.o is not found in archive
     0509-022 Cannot load module lresynclv.
     0509-150   Dependent module /usr/lib/libhmc.a(shr.o) could not
be loaded.

Install the fix for this APAR:

IV79848 MIRRORVG/SYNCVG ON MIMIMAL AND MIGRATION INSTALL FAILS
http://www.ibm.com/support/docview.wss?uid=isg1IV79848

***************************************************************
* USERS AFFECTED:
* Systems running the AIX 7200-00 Technology Level
* with bos.rte.lvm at the 7.2.0.0 or 7.2.0.1 level.
***************************************************************
* PROBLEM DESCRIPTION:
* After installation of AIX 720, if you have not installed
* the optional fileset bos.liveupdate, any mirrorvg or
* syncvg commands will fail. This will occur if you do a
* minimal or migration install.
* The command will fail with the following:
* Could not load program lresynclv:
* Dependent module liblvup.a(shr.o) could not be loaded.
* Could not load module liblvup.a(shr.o).
* System error: No such file or directory
* 0516-934 /usr/sbin/syncvg: Unable to synchronize logical
* volume
***************************************************************
* RECOMMENDATION:
* Install APAR IV79848.
* Prior to fix availability, an interim fix is available from
* either
* ftp://aix.software.ibm.com/aix/ifixes/iv79848/
* https://aix.software.ibm.com/aix/ifixes/iv79848/
* Installation of the ifix does not require a reboot.
***************************************************************