Capturing core dumps for TM1 server crash analysis

Resolving The Problem

Collecting Files for Analysis Following A TM1 Server Crash
When the TM1 server crashes, you must collect the following files for TM1 support and engineering to analyse:

• The dump file generated by your debugging tool.
- From ADPlus, the dump file is named:

ProcessID_ProcessName_2nd_chance_AccessViolation_TimeStamp.dmp

For example,
PID-492_TM1SD.EXE_2nd_chance_AccessViolation_full_ODB8_2008-03-08_10-17-59-052_01EC.dmp

For a full instructions on generating these dump files, see “Installing and Running ADPlus” later in this Technote.

• The Tm1server.log files. This file is located in the TM1 server data directory. This is a rolling file, so it is common to have multiple versions of the file in your data directory. For example, Tm1server.log, Tm1server.log.1, Tm1server.log.2, etc. Collect all Tm1server.log files for delivery to Cognos support.

• All TM1ProcessError.log files generated during the server session that crashed. When a process error log file is generated, TM1 assigns a unique name that lets you readily identify which TurboIntegrator process generated the error file and the time at which the file was created. File names are assigned using the convention TM1ProcessError_<time stamp>_<process name>.log. In this convention, <time stamp> is the time (expressed as yyyymmddhhmmss GMT) at which the file was generated and <process name> is the name of the TurboIntegrator process that caused the errors. There may be multiple TM1ProcessError.log files associated with the server session that crashed. All TM1ProcessError.log files are stored in the TM1 server data directory.

• If you are running TM1 Top, collect the TM1 Top log file. The location and name of the log file are determined by the logfile parameter to theTm1top.ini configuration file. The Tm1top.ini configuration file is located in the same directory as the TM1 Top executable file (Tm1top.exe), which is usually <TM1 install dir>\bin. If you accepted the default TM1 installation directory, the full path is C:\Program Files\Cognos\TM1\bin.

• DO NOT collect the TM1 transaction log (Tm1s.log) unless specifically requested by Cognos support.

***Ensure that adequate space is available in the nominated output location. An estimate of the space required would be to double your peak memory usage***

***By default, ADPlus puts all files in a subfolder of the installation folder. By adding the -o to the end of your start command you can nominate another location for the dump files to be saved***

Installing and Running ADPlus

ADPlus is part of the Microsoft® Debugging Tools for Windows package. You can download the most recent package at http://www.microsoft.com/whdc/devtools/debugging/default.mspx. Both a 32-bit version and a 64-bit version of Debugging Tools for Windows is available, be sure to download the version that is appropriate for your operating system. For complete details on ADPlus, including system requirements, see http://support.microsoft.com/kb/286350.

Running ADPlus Directly on the TM1 Server
Follow these steps to run ADPlus directly on the computer where the TM1 server is installed:

1. Start the TM1 server.

2. Open a command prompt window and cd to the Debugging Tools for Windows directory. The default directory on a 32-bit system is C:\Program Files\Debugging Tools for Windows (x86).

3. At the command prompt, enter adplus -crash -pn tm1sd.exe

4. Click OK ***Note a message appears that indicates a new subdirectory will be created in the Debugging Tools for Window directory. The new subdirectory, which is named Crash_Mode_DateStamp_TimeStamp, receives the dump files that are generated when the TM1 server crashes.

5. Click OK ***Note you can confirm the creation of the subdirectory in Windows Explorer. Also, note that a new command prompt window is now open on your desktop. The Microsoft Console Debugger (cdb.exe) runs in this window.

***Ensure that the user starting this command does NOT log off the machine, this will stop ADPlus from running and in some cases has crashed the tm1s.exe. See attached Technote for instructions on how ADPlus can be run as a service to prevent this***

http://www-01.ibm.com/support/docview.wss?uid=swg21686483

TM1Hang - Hang Dump Instructions

With ADPlus, you can also take a snapshot of the current state of TM1 Server. The hang dump can be useful under following conditions.
• In Task Manager, CPU usage stays '00' for the TM1 Server process for a period of time (after 15 minutes)
• TM1 Top, transactions are not progressing (after 15 minutes)

If the above conditions are not met, (example CPU comes back with 01 after 2 minutes) it is not considered as a 'true hang state', Any hang dump at this point will not contain any useful information.

In a true hang situation, you should open a PMR and provide following three files.
• Hang dump
• TM1 Server log file
• TM1 Top log file

To take a snapshot of the current server state (Hang dump), but leave the TM1 server running:

1. Start the TM1 server.
2. Open a command prompt window and cd to the Debugging Tools for Windows directory.
3. Open the Windows Task Manager and note the process ID (PID) for Tm1s.exe (if running the TM1 server as an application) or Tm1sd.exe (if running the TM1 server as a service).
4. At the command prompt, enter adplus –hang –p [PID] For example, adplus –hang –p 4925 (ADplus message appears).
5. Click OK (Windows Script host message appears)***Note that this message indicates that a new subdirectory will be created in the Debugging Tools for Window directory. The new subdirectory, which is named Hang_Mode_DateStamp_TimeStamp, receives the dump file that records the current TM1 server state.
6. Click OK. A new dump file is immediately generated, recording the current server state.
7. Open the new subdirectory in the Debugging Tools for Window directory and note the presence of a single dump file. This is the file Cognos needs to debug your server.

Monitoring the TM1 Server

If you are monitoring the TM1 server, just continue your daily activity. When the TM1 server crashes, three dump files (.dmp) are created in the subdirectory.

The dump files required by Cognos to debug your crash are:

1st - chance_AccessViolation
2nd - chance_AccessViolation

Running ADPlus Remotely from a TM1 Client

In some environments, you might want to manage and run ADPlus from the computer on which a TM1 client is running, rather than directly from the TM1 server. The following two sections describe how to set up your system to accomplish this.

Installing and Running ADPlus on the TM1 Server

To install and run ADPlus on the computer where the TM1 server is running:

1. Install the Microsoft® Debugging Tools for Windows package.

2. Create a batch file named Remoteshell.cmd in the debugging tools installation directory (C:\Program Files\Debugging Tools for Windows (x86) if you accepted the default installation directory on a 32-bit system.) This batch file should contain the following single line:

C:\Program Files\Debugging Tools for Windows (x86)\remote.exe /s "cmd.exe" remoteshell

3. From a command prompt, enter the following line:

at <2 minutes past current time, 24 hour format> C:\Program Files\Debugging Tools for Windows (x86)\remoteshell.cmd

For example, if your system clock shows 4:45 PM, enter the following line at the command prompt:

at 16:47 C:\Program Files\Debugging Tools for Windows (x86)\remoteshell.cmd

This line will run the remoteshell.cmd batch file at 4:47 PM.

4. After the specified time, run the at command from a command prompt. Do not include any parameters to the command, type only 'at'.

5. Open the Windows Task Manager and note the process ID (PID) for Tm1s.exe (if running the TM1 server as an application) or Tm1sd.exe (if running the TM1 server as a service). You will need to know the PID when running ADPlus from the TM1 client.

Running ADPlus from a TM1 Client

To run ADPlus from a computer on which a TM1 client is installed:

1. Copy the remote.exe and breakin.exe files from the debugging tools installation directory on the TM1 server computer. (C:\Program Files\Debugging Tools for Windows (x86) if you accepted the default installation directory on a 32-bit system.)

2. Save remote.exe and breakin.exe to an easily identified directory on the computer running the TM1 client. These instructions assume that remote.exe is saved in C:\debuggers.

3. Open a command prompt and cd to the C:\debuggers folder.

4. Enter the following command:

remote.exe /c <remote server name> remoteshell

remote server name is the name of the computer on which the TM1 server is installed; it is not the name of the TM1 server. For example, if smithers is the name of the computer where the TM1 server is installed, you would enter the command remote.exe /c smithers remoteshell

5. You are now running commands on the server computer from the command prompt on the client computer. To confirm that you are properly connected to the server, enter dir C:\ at the command prompt. You should see the contents of the C drive on the server.

6. From the command prompt, cd to the debugging tools installation directory on the TM1 server computer. Again, this is C:\Program Files\Debugging Tools for Windows (x86) if you accepted the default installation directory on a 32-bit system.

7. Enter the following at the command prompt:

Adplus -crash -quiet -p <PID>

PID is the process ID of the TM1 server. For example, if the process ID for the TM1 server is 1588, enter Adplus -crash -quiet -p 1588 at the command prompt. You should receive confirmation that the debugger is attached to the TM1 server. ADPlus is now ready to collect debugging information and will continue to monitor the TM1 server until it crashes.

Stopping ADPlus

Usually, you will want to let ADPlus run until a TM1 server crash occurs. However, if you need to stop ADPlus, enter the following line at the command prompt on the client computer:

breakin.exe <PID>

PID is the process ID of the TM1 server. For example, if the process ID for the TM1 server is 1588, enter breakin.exe 1588 at the command prompt to stop the debugging process.