IV23322: TEMS STOP REQUIRED DURING AIX SHUTDOWN PROCESS

IBM Tivoli Monitoring 6.2.3 Fix Pack 5 (6.2.3-TIV-ITM-FP0005)
IBM Tivoli Monitoring 6.2.3 Fix Pack 4 (6.2.3-TIV-ITM-FP0004)

APAR status

• Closed as program error.

Error description

• Environment: AIX 5.3 TEMS ITM 622 FP7
  
  
  Problem Description: After an AIX shutdwon one or more TEMS
  database files may be unusuable. One type of error is seen when
  the operations log has this
  
  KGL0001    Index file QA1CSTSH.IDX was not closed properly.
  Verify required.
  
  but the following message is absent.
  
  KGL0002    Verify of QA1CSTSH.IDX was successful.
  
  Other typical messages look like this
  
  (4FD7A39C.0001-8:kglisopn.c,609,"I_dfopen") Unable to open data
  file QA1CSTSC.DB
  
  (4FD7A39C.0002-8:kglisopn.c,370,"iopen_db") Unable to open data
  fileQA1CSTSC.DB for database QA1CSTSC
  
  (4FD7A39C.0003-8:kglkycbt.c,463,"kglky1of") iopen_db failed.
  errno = 10, file = QA1CSTSC
  
  (4FD7A39C.0004-8:kglkyof.c,243,"OpenKeyedFile") Open failed for
  QA1CSTSC, status = 5
  
  There are many other possible messages. The TEMS is unusable or
  malfunctioning badly. The common thread is a flood of messages
  which mention QA1* files.
  
  
  Detailed Recreation Procedure:
  Recreation is impractical. One estimate is that the issue
  occurs one time per 1,000 to 3,000 AIX system shutdowns.
  
  The issue can also occur in other cases like
  
  1) TEMS crash or hang or loop
  2) Mountpoint full.
  3) Other undiagnosed cases
  
  This report is only against the AIX shutdown case.
  

Local fix

• A prevention for some instances of this issue has been
  documented in a technote. This will *not* prevent all such cases
  but may prevent one common case.
  
  Stop TEMS Process during AIX Shutdown
  http://www.ibm.com/support/docview.wss?uid=swg21598557
  

Problem summary

• For all UNIX and Linux environments except AIX, the installer
  creates a script that starts and stops all of the currently
  installed components.  The installer also configures the
  operating system to run this script at both system startup and
  system shutdown.  As a result, all of the components are started
   whenever the system is started and all of the components are
  gracefully stopped when the system is shut down.
  
  For AIX environments, the installer establishes a script that
  only starts components.  It contains no logic for stopping them.
  This can cause problems for some components that do not react
  well when their process is killed directly by the operating
  system.
  

Problem conclusion

• The code was changed so that same stop logic for the other
  operating systems is included for AIX.  It also emits a new
  message telling the user how to invoke the script at system
  shutdown.
  
  Install Actions
  
  Unlike other UNIX and Linux environments, AIX does not provide a
  shutdown mechanism that can be updated by IBM Tivoli Monitoring
  directly.  So, while the installer creates a script that is
  capable of stopping all components, it has no way to tell the
  operating system to invoke it automatically at system shutdown.
  
  The AIX shutdown command runs a user-supplied script named
  /etc/rc.shutdown.  AIX runs this script before it kills running
  processes.  You must create or modify your /etc/rc.shutdown to
  do the equivalent of the following:
  
     autostart_num=$(head -n 1 CANDLEHOME/registry/AutoStart);
     /etc/rc.itm${autostart_num} stop
  
  Replace "CANDLEHOME" with the name of your installation
  directory.
  
  For more complete information about /etc/rc.shutdown, refer to
  AIX documentation on the shutdown command.
  
  
  The fix for this APAR is contained in the following maintenance
  packages:
  
     | fix pack | 6.2.3-TIV-ITM-FP0003
  

Temporary fix

• Manually implement your own script that invokes itmcmd
  incantations to stop all of the components gracefully.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  OMEG DIST INSTA

• Fixed component ID

  5608A41CI

Applicable component levels

• R623 PSY

     UP