About CICS Transaction Gateway traces and dumps

Resolving The Problem

There are a number of trace and dump options available to provide documentation to assist in problem determination with CICS TG. This document provides an overview of how to gather the documentation.

For full information and command syntax see the relevant CICS TG Information Center:

http://www.ibm.com/software/htp/cics/ctg/library/index3.html

• CICS TG traces
• Client daemon trace
• Gateway daemon trace
• JNI trace
• Java™ Client application trace
• ECI V2 Client trace
• .NET Client trace
• JCA Connector trace
• MVS System Dump of CICS TG Address Space
• Extracting Java information from an MVS System Dump
• Dr Watson dump
• CICS TG dumps
• Heap dump
• System dump
• Java dump
• JVM dump
• JVMstack dump
• CTGINFO dump
• Tracing with CICS TG in remote mode on z/OS
• Tracing with CICS TG in remote mode on Multiplatforms

CICS TG traces

This section of the document details information provided by the different traces, and how to gather them.

Where possible when submitting traces to IBM Software Support please provide both unformatted and formatted versions.

Client daemon trace

Provides:

For CICS TG on Multiplatforms only, this provides a comprehensive trace of flows through the Client daemon process, including calls made by the API, and network flows to and from CICS.

Wrapping trace:
The size of the binary trace file can be controlled by specifying that it wraps into a second trace file. Use the Client trace file wrap size (KB) configuration setting to turn on wrapping trace; specify the maximum size of the wrapping trace (in KB). If this value is set to 0, wrapping trace is not turned on.

When wrapping trace is turned on with standard I/O tracing, two files (called cicscli.bin and cicscli.wrp) are used. Each file can be up to half the size of the value of Client trace file wrap size.

Memory mapped:
If wrapping trace is enabled, memory-mapped tracing can also be enabled by specifying the -b switch with the cicscli command to turn tracing on.

Memory-mapped tracing uses the operating system's paging mechanism to swap data between memory and the trace file, significantly reducing the performance impact when compared to standard file I/O, because the trace file is opened and written to less frequently.

If the operating system itself fails, data might be lost, and the trace file might be corrupted. If you are diagnosing problems where the server fails and needs to be restarted, use standard I/O tracing instead of memory-mapped tracing.

If you use memory-mapped tracing, the size of the trace files is limited to 10 MB, and in addition to cicscli.bin and cicscli.wrp, you might see a series of files of the form cicscli.wrp1, cicscli.wrp2...cicscli.wrpn, where n is the number of files needed to hold the total amount of trace data specified in the Client trace file wrap size (KB) field of the Configuration Tool. The trace formatter finds all the files in the sequence when you format the binary files.

Enabling:

To start the Client daemon with tracing active, or start tracing when the Client daemon is already active, use the following command, which will turn on all memory-mapped CICS Client component tracing:


cicscli -s -d -m=all -b

Tracing all components might adversely affect the performance of the Client daemon. If this proves to be the case substantial diagnostic information can be collected, whilst incurring a lower performance impact. This can be achieved by specifying the following command, which enables all API, protocol driver and Client daemon trace points:


cicscli -s -d -m=api.2,ccl,drv.2 -b

Note: If the client is already running when these commands are issued the following message will be issued:

CCL8035I The Client daemon is already started
This an informational message and does not indicate an error.

Disabling:

Run the following command:


cicscli -o

The cicscli options shown have the following meanings:

-s Start the Client daemon
-d Start client tracing
-m Specify the client components to be traced
-b Use memory mapped tracing
-o Stop client tracing

Full details of the cicscli options can be found in the relevant CICS Transaction Gateway Information Center.

Trace location:

On Windows XP and 2003 with CICS TG V8.0 and later:

The default Client daemon trace location is the C:\Documents and Settings\All Users\Application Data\IBM\CICS Transaction Gateway folder.

On Windows XP and 2003 prior to CICS TG V8.0:

The default Client daemon trace location is the C:\Program Files\IBM\CICS Transaction Gateway\bin\ folder.

On Windows Vista, Windows 2008 and Windows 7:

The default Client daemon trace location is the C:\ProgramData\IBM\CICS Transaction Gateway folder.

On UNIX® and Linux:

By default the Client daemon trace is written to the /var/cicscli folder.

For more details of CICS TG product file locations see the CICS TG for Multiplatforms V8.0 Information Center topic "Location of product files" :
http://publib.boulder.ibm.com/infocenter/cicstgmp/v8r0/index.jsp?topic=/com.ibm.cics.tg.doc/ctgwin/cclahlinstfiles.html

The trace file name is cicscli.bin, and cicscli.wrpn (if wrapping trace or memory mapped trace is enabled).

Note: All cicscli.bin and cicscli.wrpn files created during a tracing run must be present for successful formatting of wrapping and memory mapped traces. When submitting traces to IBM Software Support please ensure that all the created files are submitted.

Formatting:

The following command will format a CICS Client trace with summary and detail options:


cicsftrc -i=cicscli.bin -s -d

The following command will format a CICS Client wrapping or memory mapped trace with summary and detail options:


cicsftrc -i=cicscli.bin -s -d -w

Gateway daemon trace

Provides:

Detailed information on the flow of work through the Gateway daemon, including the various pools of threads, and calls to the JNI layer.

Enabling:

Start the CICS TG as follows:


ctgstart -x

For CICS TG V6.0 and later, in remote mode, the Gateway daemon trace can be enabled dynamically using the following commands:

On Multiplatforms (from a command prompt):


ctgadmin -a trace -tlevel=4

On z/OS (from the SDSF console):


/F <cicstg_job>,APPL=TRACE,TLEVEL=4

The TLEVEL options are as follows:
0 No trace information is output.
1 Exception tracing. Only exceptions are traced.
2 Trace exceptions, and entry and exit of methods.
3 Trace exceptions, some internals, and entry and exit of methods.
4 Full debug tracing (all trace points, equivalent to ctgstart -x).

Disabling:

Restart the CICS TG without the -x parameter.

For CICS TG V6.0 and later, in remote mode, the Gateway daemon trace can be disabled dynamically using the following commands:

On Multiplatforms (from a command prompt):


ctgadmin -a trace -tlevel=0

On z/OS (from the SDSF console):


/F <cicstg_job>,APPL=TRACE,TLEVEL=0

Trace Location:

Trace location defaults to the standard error stream but can be redirected to a file if required by specifying the -tfile=pathname option on the ctgstart command, or tfile=gateway.trc in the CICS TG configuration file.

Formatting:

This trace is in text format and does not require formatting.

Full details of all the available Gateway daemon trace options can be found in the relevant CICS Transaction Gateway Information Center.

JNI trace

Provides:

Detailed information on the flow of work through the CICS TG JNI layer. With CICS TG for Multiplatforms, the JNI is a thin layer and the JNI trace does not show a large amount of information. With CICS TG for z/OS, the JNI trace provides details of calls made to CICS and other resources used by the CICS TG.

Enabling in remote mode:

Add the following environment variables to the environment that the CICS TG calls are made from (if running on z/OS in the STDENV file) before starting the CICS TG:


CTG_JNI_TRACE=<filename>

CTG_JNI_TRACE_ON=YES

or

Start the CICS TG, using the command:


ctgstart -j-Dgateway.T.setJNITFile=filename


where filename is the name of the file to which trace output is to be sent.

or

For CICS TG for z/OS V6.0 and later, the JNI trace can enabled using the following MVS system command from the SDSF console:


/F <cicstg_job>,APPL=TRACE,JNILEVEL=1

Enabling in local mode:

Set the following environment variables in the J2EE server environment:


CTG_JNI_TRACE=<filename>

CTG_JNI_TRACE_ON=YES

or
Java Client applications can be launched specifying the system property gateway.T.setJNITFile, as shown in the following example:


java -Dgateway.T.setJNITFile=filename application

where:
filename is the name of the file to which trace output is to be sent.
application is the application to launch.

Disabling:

Restart the CICS TG or WAS environment without the environment variables set. For CICS TG V6.0 for z/OS and later the JNI trace can be disabled using the following MVS system command from the SDSF console:

/F <cicstg_job>,APPL=TRACE,JNILEVEL=0

Trace location:

The trace location defaults to the standard error stream but can be redirected to a file if required.

Formatting:

This trace is in text format and does not require formatting.

Java Client application trace

Provides:

Detailed information on the flow of work through the CICS TG base API classes. In remote mode it shows network flows sent to and received from the Gateway daemon. In local mode this shows the complete flow of requests through the CICS TG Java components and calls to the Java Native Interface (JNI).

Enabling:

Java Client application trace can be enabled programmatically or via Java directives passed through to the JVM that the application runs in. We recommend using Java directives because this means no changes need to be made to the application.

Java Directive

Add the following to the Java environment:

-Dgateway.T=on -Dgateway.T.setTFile=filename

For example to turn on Java Client application trace for the sample Java application EciB1 to be written to file EciB1trace.txt the following command could be used:

java -Dgateway.T=on -Dgateway.T.setTFile=EciB1trace.txt com.ibm.ctg.samples.EciB1

Programmatically

Add the following statements to your Java application:

T.setDebugOn(true);

T.setTFile("filename");

Disabling:

Remove the directives or statements added above from the JVM or the application as appropriate.

Trace location:

Trace location defaults to the standard error stream but can be redirected to file if the filename directive or statement shown above are used.

Formatting:

This trace is in text format and does not require formatting.

Full details of all the available Java client application trace options can be found in the CICS Transaction Gateway Information Center.

ECI V2 Client trace

Provides:

Detailed information on the flow of work through the CICS TG ECI V2 client application.

Enabling:

Programmatically

ECI V2 Client trace can be enabled programmatically by coding:

CTG_setAPITraceLevel(CTG_TRACE_LEVELn);


in the client application.

Environment Variables

By specifying the following environment variable before the ECI V2 client application is started:

CTG_CLIENT_TRACE_LEVEL=n

where n is a value in the range 0 to 4, matching the trace levels
specified in the API.

0 = Disable all trace
1 = Exception trace
2 = Event trace
3 = Entry/exit trace
4 = Debug trace

Disabling:

Programmatically

ECI V2 client trace can be disable programmatically by coding:
CTG_setAPITraceLevel(CTG_TRACE_LEVEL0);

Environment Variables

The ECI V2 client trace cannot be disabled using environment variables,

Trace location:
The default trace destination is stderr, but this can be overridden programmatically or using environment variables.

Programmatically

The trace file can be specified programmatically by coding:

CTG_setAPITraceFile("filename");

Environment Variables

By specifying the following environment variable before the ECI V2 client application is started: 
CTG_CLIENT_TRACE_FILE=filename

where filename is the name of the file that the trace will be written to.

Formatting:

This trace is in text format and does not require formatting.

.Net Client Trace

Provides:

Detailed information on the flow of work through the CICS TG .NET client application.

Enabling:

Programmatically

Using the Trace class. TraceLevel sets or gets the trace level.

Application Configuration File

Trace can be enabled using the CtgTrace trace switch in an application configuration file (an XML file). The switch allows the trace to be specified as an IBM.CTG.TraceLevel value, a System.Diagnostics.TraceLevel value, or an integer between 0 and 4 inclusive. In the following example the switch value="CtgTrcLevel4" specifies Level 4 tracing:

<configuration>
<system.diagnostics>
<switches>
<add name="CtgTrace" value="CtgTrcLevel4" dataDumpOffset="0" dataDumpLength="128"/>
</switches>
<trace>
<listeners>
<add name="AppTrace" type="System.Diagnostics.TextWriterTraceListener" initializeData="Application.trc"/>
</listeners>
</trace>
</system.diagnostics>
</configuration>

The following levels of tracing are available:

CtgTrcDisabled (or 0) Disable all trace
CtgTrcLevel1 (or 1) Exception trace
CtgTrcLevel2 (or 2) Event trace
CtgTrcLevel3 (or 3) Entry/exit trace
CtgTrcLevel4 (or 4) Debug trace

Integer values 0 -4 can, corresponding to the above CtgTrcxxxx levels can also be specified.
where n is a value in the range 0 to 4, matching the trace levels

Disabling:

Programmatically

Set TraceLevel to CtgTrcDisabled or 0.

Trace location:

Application Configuration File

The .Net Client trace is written to the file specified in the application configuration file, in the name="AppTrace" type="System.Diagnostics.TextWriterTraceListener" initializeData="" definition. See the example above.

Formatting:

This trace is in text format and does not require formatting.

Full details of all the available .Net Client trace options can be found in the CICS Transaction Gateway Information Center.

JCA Connector trace

Provides:

Detailed information on the flow of work through the deployed connector, which allows the support teams to verify the interactions between the Connector and WebSphere, and the Connector and the "base" CICS TG classes.

Enabling:

Enable trace in WebSphere following the instructions for your version of WAS. Add the following to the trace components list for WAS:

WAS.j2c=all=enabled:com.ibm.connector2.*=all=enabled

In the Connection Factory Custom properties for the CICS TG resource adaptor, set the tracelevel property to 3.

Disabling:

Remove the trace options added above from the standard WebSphere tracing console.

Trace location:

JCA Connector trace is integrated with the WebSphere tracing mechanism and will be written to the same location as other WebSphere traces. Typically this is trace.log.

Formatting:

This trace is in text format and does not require formatting.

MVS System Dump of CICS TG Address Space

To obtain an MVS system dump of CICS TG:

1. To display OMVS tasks. On the SDSF console use the command:

/D OMVS,A=ALL

2. Find the CICS TG task, and note the ASID.

3. Enter the DUMP command with a suitable comment. For example:

/DUMP COMM=(CICS TG DUMP)

4. Reply to the message with the ASID as follows:

/R nn,ASID=aa,END

where nn is the message number for the reply, and aa is the ASID.

Provides:
An MVS system dump of the CICS TG address space.

Location:
MVS system dump dataset.

Formatting:
Viewed using IPCS.

The IBM Memory Analyzer Tool can be used to investigate JVM information contained in an MVS system dump. If the Gateway daemon is running with Java V6.0 or earlier, the system dump must be pre-processed using the Java dump extractor tool, jextract. The jextract tool must be run on a system running the same level of Java as that of the system that produced the system dump. Dumps for Gateway daemons running with Java V6.0.1 or later do not need to be pre-processed using jextract.
See the Java documentation for more details of the jextract tool.

Extracting Java Information from an MVS System Dump

The IBM Memory Analyzer Tool can be used to investigate JVM information contained in an MVS system dump. If the Gateway daemon is running with Java V6.0 or earlier, the system dump must be pre-processed using the Java dump extractor tool, jextract. The jextract tool must be run on a system running the same level of Java as that of the system that produced the system dump. MVS system dumps for Gateway daemons running with Java V6.0.1 or later do not need to be pre-processed using jextract.
See the Java documentation for more details of the jextract tool.

Dr Watson dump

Microsoft Windows provides an error debugging tool that might produce diagnostic dumps relating to CICS TG if an error is detected while CICS TG is running.

For details of Dr Watson please refer to the Microsoft web page:

http://support.microsoft.com/kb/308538

CICS TG dumps

In CICS TG V6.0 and later the following commands can also be used to collect dumps additional dumps.

On z/OS:
/F <jobname>, APPL=DUMP,<option>



On distributed platforms:
ctgadmin -a dump -<option>



Options:

-heap	Request a Java heap dump from the Gateway daemon.
-system	Request a system dump from the Gateway daemon.
-java	Request a Java core dump from the Gateway daemon.
-jvm	Request a JVM information dump from the Gateway daemon.
-jvmstack	Request a JVM stack dump from the Gateway daemon.
-ctginfo	Request a CICS TG information dump from the Gateway daemon.

For example (requests both a Java heap and a system dump):

On z/OS:
/F <jobname>,APPL=DUMP,HEAP,SYSTEM

On distributed platforms:
ctgadmin -a dump -heap -system

Heap dump

Available only with the IBM JVM.

Provides:
A Java heap dump to the default location. The dump file has a file name of the form heapdump.YYYYMMDD.HHMMSS.<pid>.phd

Formatting:
Required.

Location:
See Dump locations below.

Back to CICS TG Dumps

System dump

Available only with the IBM JVM

Provides:
A Java system dump to the default location. The dump file has a file name of the form core.YYYYMMDD.HHMMSS.<pid>.dmp

Formatting:
Required.

Location:
See Dump locations below.

Back to CICS TG Dumps

Java dump

Available only with the IBM JVM

Provides:
A Java dump to the default location. The dump file has a file name of the form javacore.YYYYMMDD.HHMMSS.<pid>.txt

Formatting:

This trace is in text format and does not require formatting.

Location:
See Dump locations below.

Back to CICS TG Dumps

JVM dump

Provides:
A dump containing current JVM memory usage.

Formatting:

This trace is in text format and does not require formatting.

Location:
See Dump locations below.

Back to CICS TG Dumps

JVMstack dump

Provides:
A dump containing only the Java call stack.

Formatting:

This trace is in text format and does not require formatting.

Location:
See Dump locations below.

Back to CICS TG Dumps

CTGINFO dump

Provides:
A dump containing information about the configuration of the CICS TG.

Formatting:

This trace is in text format and does not require formatting.

Back to CICS TG Dumps

CICS TG dump locations

Location of CICS TG dump files on z/OS:

• The dumps write information to a location chosen from the first available option in the following list:
• The location specified by the _CEE_DMPTARG environment variable, if set.
• The current working directory of the JVM processes, if permitted.
• The location specified by the TMPDIR environment variable, if set.
• The /tmp directory.
• If the dump cannot be written to any of the previous locations, it is sent to STDERR.

Location of CICS TG dump files on distributed platforms:

• The dumps write information to a location chosen from the first available option in the following list:
• The location specified by the IBM_JAVACOREDIR environment variable, if set.
• The current working directory of the JVM processes, if permitted.
• The location specified by the TMPDIR environment variable, if set.
• The /tmp directory.
• If the dump cannot be written to any of the previous locations, it is sent to STDERR

Tracing with CICS TG in remote mode on z/OS

Tracing with CICS TG in remote mode on multiplatforms