Collecting data for IBM FileNet Image Services (IS) 4.x or P8 PE (Process Engine) for AIX

Answer

Collecting data for IBM FileNet IS 4.x or P8 PE (Process Engine) for AIX

1. Initial Triage

• What products, versions, and system configuration exist? The more detail the better.
  Example: One Root/Index server, with a BES server, running IS 4.0 Service Pack 5 Fix Pack 9, Oracle 9i (9.2.0.x), AIX 5.3, etc.
  
• What is the error message or symptom? If available, screen shots or log file.
  
• Which system is affected? Production, QA (Quality Assurance), Development, DR (Disaster Recovery) or Test?
  
• What changes have been made to the environment recently?
  Example: hardware or software added or modified? change in network configuration? users added?
  
• Has this problem occurred before? If so, how often and can it be reproduced?
  
• Is there a workaround?
  
• What is the current business impact?

• 911 completes:

1. Run the 911 prior to restarting the IS software
2. If there are multiple IS Servers (Root/Index server, OSAR/MSAR servers, and Application servers), run 911 on each server and collect the data from each server prior to restarting the IS software.

• 911 hangs:
  
1. If 911 hangs, then enter a Ctrl+C to quit the currently running 911.
2. Execute the following commands to gather the logs (do this prior to restarting IS software):
• cd /fnsw/local/tmp
• script info.<date>.txt (Example: script info.061708.txt)
• cormon -p
• ipcs
• ipc_tool -Ni |tee
• ipc_tool -s |tee
• ipc_tool -t |tee
• vmstat 5 5
• iostat 5 5
• ipc_tool -Ni |tee (2nd time to run this)
• ps -ealf
• MKF_debug
• netstat -a
• netstat -in
• dbp -s
• cormon -p (2nd time to run this)
• PPMOI (tp, svc, l, pre are in IS 4.1.2)
• rp
• qp
• sh
• st
• tp
• svc
• l
• pre
• quit
• fn_util whichfn
• od -c /fnsw/procs/*
• df
• lsps -a
• no -a
• netstat -m
• netstat -p tcp
• ^D (Control+D)
3. Get the current modules stamp information:
   getstamps.sh -stamps
4. Get the highest numbered CDB file from the Root/Index server in /fnsw/local/sd/conf_db/IMS_###.cdb

• The 911 log file from/fnsw/local/logs/911/911.<date>.<sequence number>.{tar}.Z. This would be a compressed tar file containing all the data gathered by running the 911 script. Depending on your 911 version, the tar suffix may or may not be part of the output file name.
• Get the WAL log if it exists from /fnsw/client/logs/wal<yyyymmdd>
• spacerpt > space.06172008.txt
• Optionally, run the performance reports after restarting the IS software.
• perf_report -a -csv -m3000
• perf_report -rful -m3000 > perf.rful<date>.txt
• errpt -a > errpt.txt
• lsvg > lsvg.txt
• lsvg rootvg >> rootvg.txt
• lsvg -l rootvg >> rootvg.txt
  Repeat for any other Volume Groups (VG). Use the VG name. (Example: lsvg fnvg >> fnvg.txt).
• lslpp -l > lslpp.txt
• Gather the output file (e.g., /fnsw/local/tmp/info.<date>.txt) or script file you created and other data to IBM. May want to compress or gnu zip (gzip) the output file.
  Example: "compress info.06172008.txt"
• Send the 911 output file and other data to IBM.
  
  

1. IBM Support Assistant (ISA)
2. Electronic Service Request (SR)
3. FTP to the Enhanced Customer Data Repository (ECuRep)