Data collected as part of FODC

Files containing diagnostic information that are collected by FODC

FODC collects diagnostic information from a number of sources. The exact diagnostic information captured by FODC depends on the type of problem encountered and might include:

Administration notification log (instance_name.nfy)

• Operating system: All
• Default location:
  • Linux and UNIX: Located in the directory specified by the diagpath database manager configuration parameter.
  • Windows: Use the Event Viewer Tool (Start > Control Panel > Administrative Tools > Event Viewer)
• Created automatically when the instance is created.
• When significant events occur, DB2® writes information to the administration notification log. The information is intended for use by database and system administrators. The type of message recorded in this file is determined by the notifylevel configuration parameter.
  Note: When the diagsize database manager configuration parameter is set to a nonzero value, the single administration notification log file behavior (instance_name.nfy) will be changed to a rotating log behavior (instance_name.N.nfy).

DB2 diagnostic log (db2diag.log)

• Operating system: All
• Default location: Located in the directory identified by the diagpath database manager configuration parameter.
• Created automatically when the instance is created.
• This text file contains diagnostic information about error and warnings encountered by the instance. This information is used for troubleshooting and is intended for technicians at IBM Software Support. The type of message recorded in this file is determined by the diaglevel database manager configuration parameter.
  Note: When the diagsize database manager configuration parameter is set to a nonzero value, the single diagnostic log file behavior (db2diag.log) will be changed to a rotating log behavior (db2diag.N.log).

DB2 administration server (DAS) diagnostic log (db2dasdiag.log)

• Operating system: All
• Default location:
  • Linux and UNIX: Located in DASHOME/das/dump, where DASHOME is the home directory of the DAS owner
  • Windows: Located in "dump" folder, in the DAS home directory. For example: C:\Program Files\IBM\SQLLIB\DB2DAS00\dump
• Created automatically when the DAS is created.
• This text file contains diagnostic information about errors and warnings encountered by the DAS.

DB2 event log (db2eventlog.xxx, where xxx is the database partition number)

• Operating system: All
• Default location: Located in the directory specified by the diagpath database manager configuration parameter
• Created automatically when the instance is created.
• The DB2 event log file is a circular log of infrastructure-level events occurring in the database manager. The file is fixed in size, and acts as circular buffer for the specific events that are logged as the instance runs. Every time you stop the instance, the previous event log will be replaced, not appended. If the instance traps, a db2eventlog.XXX.crash file is also generated. These files are intended for use by IBM Software Support.

DB2 callout script (db2cos) output files

• Operating system: All
• Default location: Located in the directory specified by the diagpath database manager configuration parameter
• If db2cos scripts are executed as a consequence of an FODC outage, db2cos output files will be placed under the FODC directory that was created in the location specified by the diagpath database manager configuration parameter.
• Created automatically when a panic, trap or segmentation violation occurs. Can also be created during specific problem scenarios, as specified using the db2pdcfg command.
• The default db2cos script will invoke db2pd commands to collect information in an unlatched manner. The contents of the db2cos output files will vary depending on the commands contained in the db2cos script, such as operating system commands and other DB2 diagnosing tools. For more details on the tools that are executed with the db2cos script, open the script file in a text editor.
• The db2cos script is shipped under the bin/ directory. On UNIX, this directory is read-only. To create your own modifiable version of this script, copy the db2cos script to the adm/ directory. You are free to modify this version of the script. If the script is found in the adm/ directory, it is that version that is run. Otherwise, the default version in the bin/ directory is run.

Dump files

• Operating system: All
• Default location: Located in the directory specified by the diagpath database manager configuration parameter
• If these files are dumped during an FODC outage, they will be placed under the FODC directory.
• Created automatically when particular problem scenarios arise.
• For some error conditions, extra information is logged in binary files named after the failing process ID. These files are intended for use by IBM Software Support.

Trap files

• Operating system: All
• Default location: Located in the directory specified by the diagpath database manager configuration parameter
• If these files are dumped during an FODC outage, they will be placed under the FODC directory.
• Created automatically when the instance ends abnormally. Can also be created at will using the db2pd command.
• The database manager generates a trap file if it cannot continue processing due to a trap, segmentation violation, or exception.

Core files

• Operating system: Linux and UNIX
• Default location: Located in the directory specified by the diagpath database manager configuration parameter
• If these files are dumped during an FODC outage, they will be placed under the FODC directory.
• Created by the operating system when the DB2 instance terminates abnormally.
• Among other things, the core image will include most or all of the memory allocations of DB2, which may be required for problem descriptions.

FODC package path and contents

FODC creates the FODC package directory in the FODC path specified. You specify the FODC path through the FODCPATH registry variable setting or the db2fodc -fodcpath command parameter option. If you do not specify any FODC path, first occurrence data capture sends diagnostic information to the current diagnostic directory path (diagpath or alt_diagpath). A db2diag log file diagnostic message is logged to identify the directory name used for FODC. The capture of diagnostic information can generate a significant volume of diagnostic data, depending on what parameters are specified, and enough space must be available in the directory path where FODC stores diagnostic information. To avoid a scenario where FODC fills all the available space in the file system and impacts your data server, it is recommended that you specify a FODC path where FODC can store the diagnostic data.

For automatic FODC, a package is collected for the member or partition where the problem is occurring; if the problem is occurring on multiple members, multiple packages are collected in separate FODC package directories. The FODC package directory follows the naming convention FODC_outageType_timestamp_member_number, where outageType is the problem symptom, timestamp is the time of FODC invocation, and member_number is the member or partition number where the problem occurred. For example, when a trap occurs on member 1, FODC might automatically create a package named like FODC_Trap_ 2010-11-17-20.58.30.695243_0001.

For manual FODC, a package is collected for the member(s) or partition(s) you specify. The naming convention for the FODC package directory is FODC_manualOutageType_timestamp_memberList, where manualOutageType is the problem symptom, timestamp is the time of FODC invocation, and memberList is a list of the members or partitions where the problem occurred. For example, the manually issued command db2fodc -hang -basic -member 1,2,3 -db sample creates a manual FODC package for members 1,2 and 3, and might be named like FODC_hang_ 2010-11-17-20.58.30.695243_0001.0002.0003.

FODC sends the following diagnostic information to the FODC package directory:

db2fodc -clp collects the following information:

• Operating system information
• Instance and database configuration information

db2fodc -hang collects the following information:

• Basic operating system information. The problem could be due to OS level, patches, etc.
• Basic DB2 configuration information.
• Operating system monitor information: vmstat, netstat, iostat, etc.
  • 2 iterations at least: with timestamps saved
• Partial call stacks: DB2 stack traces of top CPU agents.
• Operating system trace: trace on AIX®.
• Diagnostic information collected by db2pd.
• DB2 trace.
• Full DB2 call stacks.
• Second round of DB2 configuration information.
  • Including second DB2 trace collection.
• Snapshot information: db2 get snapshot for database, applications, tables, and so on.
  • Information will be collected per node in case of multiple logical nodes.

db2fodc -indexerror collects the following information:

• Basic Mode
  • db2cos_indexerror_short(.bat) script is run. See script for additional details.
  • If applicable db2dart commands exist in the script, the db2dart /DD and/or db2dart /DI data formatting actions are run with number of pages limited to 100.
• Full Mode
  • db2cos_indexerror_short(.bat) and db2cos_indexerror_long(.bat) scripts are run. See scripts for additional details.
  • If applicable db2dart commands exist in the script db2cos_indexerror_short(.bat), the db2dart /DD and/or db2dart /DI data formatting actions are run with number of pages limited to 100.
  • If applicable db2dart commands exist in the script db2cos_indexerror_long(.bat), the db2dart /DD and/or db2dart /DI data formatting actions are run with no limit to the number of pages.
  • If applicable db2dart commands exist in the db2cos_indexerror_long(.bat) script, the db2dart /T command is run. This command requires the database be offline.

db2fodc -perf monitors the system possibly collecting the following information:

• Snapshots
• Stacktraces
• Virtual Memory (Vmstat)
• Input/Output information (Iostat)
• traces
• Some other information depending on the case. See the script for more details.

db2fodc -preupgrade collects the following information:

• Operating system information
• Instance and database configuration information, such as output of the db2level command, environment variables, output of the db2 get dbm cfg command, and the db2nodes.cfg file
• System catalog data and statistics, such as optimizer information collected by the db2support -d dbname -c -s -cl 0 command
• Operating system monitoring data, such as output of the netstat -v and ps -elf commands
• System files
• Package information, as returned by the DB2 LIST PACKAGES FOR SCHEMA schema-name SHOW DETAIL command for all schema names
• Any FODC_Preupgrade directories found in db2dump/. These directories contain information such as performance data, top dynamic SQL queries, and explain plans
• The logfile from the db2ckupgrade command in /tmp/db2ckupgrade.log.processID, if it exists

The following diagnostic information is also included when you specify the members on which to collect:

• Snapshots (after turning on all monitor switches)
• The db2pd command output for the -everything, -agents, -applications, -mempools, and -fcm parameters
• The dynamic SQL statements used most frequently
• The query plans for SQL statements
• The explain plans for static packages

Manual db2fodc command invocation results in the creation of a log file named db2fodc_symptom.log in the FODC_symptom directory, where symptom is one of the collection types, such as hang or perf. Inside this file, the db2fodc command also stores status information and metadata describing the FODC package inside the FODC subdirectory. This file contains information on the type of FODC, the timestamp of the start and end of data collection, and other information useful for the analysis of the FODC package.