IV04047: SHLAP64 UNABLE TO PROCESS ORACLE REQUEST LEADING TO KERNEL HANG APPLIES TO AIX 6100-06

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• The oracle RAC instance can not be connected.
  From the nmon's top-processes interface, it displayed
  shlap64 cost lots of CPU resource.
  
  This issue occurs for 64 bit process trying to load
  library privately.
  
  (0)> lq
                      BUCKET HEAD            COUNT
  
  slist_table+0003C0     61 pvthread+1850800   274
  
  
  (0)> th pvthread+1024F00
                  SLOT NAME     STATE    TID PRI   RQ CPUID
   CL  WCHAN
  
  pvthread+1024F00 66127 sh       SLEEP 24F231 03C   19
      0
  F10006004593EF20
  
  NAME................ sh
  WTYPE............... WEVENT
  .................tid :000000000024F231  ......tsleep
  :FFFFFFFFFFFFFFFF
  ...............flags :00000000  ..............flags2
  :00000000
  
  ....
  ....
  
  (0)> f 66127
  pvthread+1024F00 STACK:
  [000667F0]e_block_thread+000290 ()
  [00067448]e_sleep_thread+0000E8 (??, ??, ??)
  [001B9D58]shlap_request+000118 (??, ??)
  [001BAB3C]shlap_rr+00003C (??, ??, ??, ??)
  [00657350]ld_lib_getsvcfd+0002D0 (??, ??, ??)
  [00656AD0]ld_getsvcfd+000070 (??, ??)
  [006564CC]ld_privinstance+00036C (??, ??)
  [00659354]ld_priv_instances+0002B4 (??, ??, ??)
  [00654800]process_scc+000260 (??, ??)
  [0065449C]search+0001FC (??, ??)
  [00659504]ld_libld_scc+0000E4 (??, ??)
  [0065AB64]ld_read+000184 (??, ??, ??, ??, ??)
  [0065B2E8]ld_libraries0+000048 (??, ??)
  [006F0724]ld_execload+0014C4 (??, ??, ??, ??)
  [003DA9BC]execvex_common+00189C (??, ??, ??, ??, ??)
  [003DDCBC]execve+0000A8 ()
  [00003850]ovlya_addr_sc_flih_main+000130 ()
  [kdb_get_virtual_memory] no real storage @ 2FF21F60
  [1001B904]1001B904 ()
  [kdb_read_mem] no real storage @ FFFFFFFFFFF9750
  (0)>
  
  
  shlap64 thread shows bad IAR:
  0)> f
  pvthread+1000E00 STACK:
  WARNING: bad IAR: 100000650, display stack from LR:
  1000006C4
  [1000006C4]00000001000006C4 ()
  [kdb_get_virtual_memory] no real storage @
  FFFFFFFFFFFF9E0
  [1000006C4]00000001000006C4 ()
  [kdb_read_mem] no real storage @ FFFFFFFFFFF9100
  

Local fix

• Change the perimission of library with chmod o+r to allow
  public loading to avoid the issue.
  

Problem summary

• kernel hang , when 64 bit process trying to load
  library privately.
  

Problem conclusion

• protecting loadlist with a loadlist lock.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  IZ86572

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  AIX 610 STD EDI

• Fixed component ID

  5765G6200

Applicable component levels

• R610 PSY U843098

     UP11/10/15 I 1000

PTF to Fileset Mapping

• U843098 bos.mp64 6.1.6.16