IBM Support

How to collect Packet Traces and other TCP/IP related traces on z/OS

Question & Answer


Question

How do I collect a Packet trace on z/OS? Or any other traces related to the TCP/IP stack?

Answer

Table of Contents

A. General Considerations for using an External Writer
B. Collecting Multiple Traces
C. General Considerations for using internal buffers

D. Collecting a Packet (SYSTCPDA) Trace
E. Collecting an Event (SYSTCPIP) Trace
F. Collecting an OSA Network Analysis (SYSTCPOT) Trace
 


A. General Considerations for using an External Writer

To collect a CTRACE using an external writer, a writer proc needs to be created in either SYS1.PROCLIB or in a library concatenated in the MASTER JCL. The name is arbitrary, but the member name and the name on the PROC statement need to match. Following is a sample writer proc:


   //CTWTR1   PROC
   //IEFPROC  EXEC PGM=ITTTRCWR  
   //TRCOUT01 DD DSNAME=USERID.TRACE.DATA,UNIT=SYSDA,
   //            SPACE=(CYL,(500,0),,CONTIG),DISP=(NEW,CATLG),DSORG=PS

The DSNAME specification can be set to any name that can be cataloged in your environment. Rather than specifying a single data set name, it would be better to have the procedure create a new unique name each time it is used. Specify the name by using system symbols (such as &SYSNAME, &LYYMMDD (date), or &LHHMMSS (time)), or define a GDG which results in a new generation each time. The space allocated needs to be representative of the amount of activity you plan on collecting.

Before you start the external writer proc, make sure that its dispatching priority is at least as high as the ASIDs being traced (or in the same WLM service class).  This action is to prevent lost records or missing trace records. Use of larger trace buffer sizes reduces the number of lost records, for more information on this topic see the General Considerations for Internal Traces. Also, make sure that the external writer data set allocation is not restricted by SMS rules or another storage manager.

For more information about the external writer and specification of the associated data sets, see the MVS Diagnosis: Tools and Service Aids manual.

The general sequence to use is as follows. The START and STOP commands cannot be used for the writer proc, you must use the TRACE WTRSTART and TRACE WTRSTOP commands. The stackname specification is generally the name of the TCPIP started proc, except as noted in the specific component sections.
  1. Start the external writer

    TRACE CT,WTRSTART=writerproc
  2. Start the CTRACE. The required reply attaches the external writer so that it can be used to write the CTRACE records in addition to any other applicable options needed.

    TRACE CT,ON,COMP=SYSTCPxx,SUB=(stackname)
    R xx,WTR=writerproc,other_options,END
  3. Some traces require other commands to be entered here, see the following sections for the type of trace being performed.
  4. Optionally verify that the trace started successfully.

    DISPLAY TRACE,COMP=SYSTCPxx,SUB=(stackname)
  5. Recreate the problem.
  6. If a command was needed at step 3, then that needs to be disabled here.
  7. Stop the CTRACE and disconnect the writer

    TRACE CT,ON,COMP=SYSTCPxx,SUB=(stackname)
    R xx,WTR=DISCONNECT,END
  8. Stop the component trace

    TRACE CT,OFF,COMP=SYSTCPxx,SUB=(stackname)
  9. Stop the external writer

    TRACE CT,WTRSTOP=writerproc,FLUSH
 


B. Collecting Multiple Traces with an External Writer
Multiple traces can be collected at the same time with external writers, which can include multiple types of traces(components) or from multiple sources (subcomponents).

If a low volume of activity is anticipated, all of the traces can all be directed to a single external writer procedure. Start the writer before any of the traces, and stop all of the traces before you stop the writer.

But if higher volumes of data are expected (or if you are uncertain), it would likely be best to direct each to its own writer procedure. In this case, ideally the data sets used by these writers need to be directed to separate DASD volumes.
 


C. General Considerations for using internal buffers

Before you collect a trace that uses the internal buffers as a part of a system dump, the buffers need to be created with enough space allocated. The size created when TCPIP starts is set by the CTIxxxyy PARMLIB member listed in the following table.
Type SUB PARMLIB Member Specification Location
Event Trace (SYSTCPIP) TCPIP name CTIEZByy BUFSIZE(xxxx) HVCOMMON
Event trace for TN3270 TN3270 server name CTIEZBzz BUFSIZE(xxxx) TN3270 private
Packet Trace (SYSTCPDA) TCPIP name (None) Twice the SYSTCPIP BUFSIZE HVCOMMON
Intrusion detection (SYSTCPIS) TCPIP name CTIIDSyy BUFSIZE(xxxx) HVCOMMON
OSA Network (SYSTCPOT) TCPIP name CTINTAyy BUFSIZE(xxxx) HVCOMMON
Dynamic routing
(SYSTCPRT)		 OMPROUTE name CTIORAyy BUFSIZE(xxxx) OMPROUTE private
Resolver (SYSTCPRE) RESOLVER name CTIRESyy BUFSIZE(xxxx) RESOLVER private
IKE trace (SYSTCPIK) IKED name CTIIKEyy BUFSIZE(xxxx) IKED private

The current setting can be verified by using the following command:

DISPLAY TRACE,COMP=SYSTCPxx,SUB=(tcpipname)

The steps to collect the trace are the same as listed for using an external writer, except that:
  • Steps 1, 7, and 9 are not performed.
  • Do not specify the WTR keyword and value in step 2.
To modify the size of the trace buffer, replace the ON keyword in the TRACE command to start the CTRACE with the desired size, for example:

TRACE CT,nnnM,COMP=SYSTCPxx,SUB=(tcpipname)

Where nnn is the new buffer size in megabytes. The buffer size is subject to the minimum and maximum buffer size established for each component.

The TCPIP trace buffers that are in HVCOMMON (64-bit addressing) and are included in all system dumps that have common storage.  Those traces that are listed as being in private need to have the corresponding address space included in the dump.
NOTE: In z/OS 1.12 and older releases, the TCPIP trace data is written to the TCPIPDS1 data space associated with the TCPIP address space. To have the buffers included in the dump, add DSPNAME=('tcpipname'.TCPIPDS1) in the reply to the DUMP or SLIP command.

This method of writing trace data to internal trace buffers might result in lost trace data as the occurrence of wrapping is possible. You should issue the dump command soon after the problem happens, or collect the dump using a trap or SLIP provided by the support center.
 


D. Collecting a Packet (SYSTCPDA) Trace
See the general directions for using an external writer or for internal buffers.  Do not specify any OPTIONS, JOBNAME, or ASID values in the reply to the TRACE command for the SYSTCPDA component.
For step 3, you need to start the Packet Trace processing in TCPIP.
VARY TCPIP,tcpipname,PKTTRACE,ON

The previous command causes all IP traffic to be traced. Add specification of one or more filters to reduce the volume of data recorded.

There are many filters that can be specified on the PKTTRACE command, the complete list can be seen in the IP System Administrator's Commands manual. The most common filters to use are:
  • An IP address (IPADDR=aa.bb.cc.dd) or a masked address (IPADDR=aa.bb.cc.0/24 or IPADDR=aa.bb.cc.0,SUBNET=255.255.255.0). Typically the specified address is for the remote system being traced.
  • A port number (PORTNUM=nnn). Typically the specification is for a server port, either local or remote.
    NOTE: For z/OS 1.8 or earlier, PORTNUM is not available. In this case, an ORed combination of SRCPORT and DESTPORT is needed (see the last example).
  • Abbreviated packet tracing (ABBREV or ABBREV=len), which reduces the amount of data recorded per packet. This specification is appropriate when you are diagnosing network or performance issues, but is not appropriate when the full set of application level data needs to be traced.
  • Discarded packets are not traced or flagged by default. Specify the DISCARD=* option to see both normal and discarded packets, or DISCARD=ALL to see only the discarded packets.

NOTE: Any changes made to PKTTRACE settings affect the records that applications that use the NETMONITOR PKTTRCSERVICE receive. Therefore, you might want to modify the filters to allow the application to continue to work while you are collecting diagnostic data (or shut down the application during this period).  Change the settings back to what they were once the diagnostics are completed (including doing a TRACE CT,ON,... command).

When you are specifying multiple filters, the ones provided on a single PKTTRACE command are ANDed together (they all have to be true for the packet to be traced). The filters can be ORed together by specifying each on a series of PKTTRACE commands.
  • The following example logically ANDs the two parameters:

    VARY TCPIP,tcpipproc,PKTTRACE,ON,IPADDR=10.9.8.7,PORTNUM=23
  • The following example logically ORs the two parameters:

    VARY TCPIP,tcpipproc,PKTTRACE,ON,PORTNUM=21
    VARY TCPIP,tcpipproc,PKTTRACE,ON,PORTNUM=20
  • The following example is specifying a port filter without using the PORTNUM keyword (do not use SRCPORT or DESTPORT unless directed or if the system is running z/OS 1.8 or earlier):

    VARY TCPIP,tcpipproc,PKTTRACE,ON,SRCPORT=53
    VARY TCPIP,tcpipproc,PKTTRACE,ON,DESTPORT=53
For step 6, use the following commands:

VARY TCPIP,tcpipproc,PKTTRACE,OFF
VARY TCPIP,tcpipproc,PKTTRACE,CLEAR

 


E. Collecting an Event (SYSTCPIP) Trace
See the general directions for using an external writer or for internal buffers. There is no command needed for step 3.

There are many possible options that can be specified in the reply to the TRACE command, which are listed in the IP Diagnosis Guide. In general, the IBM Support team provide the set to be collected. An exception is the SOCKAPI option, which is useful for debugging applications that use Sockets Macro Extended API or one of its derivatives (EZASOKET calls, CICS Sockets API (Call format or C sockets), or IMS Sockets).

The IPADDR option will filter records collected to only the ones associated with the remote system addresses or subnets listed. Specifying a local address results in no records being collected (except for local-to-local connections).

The PORT option will filter records collected to only the ones associated with the local ports listed. Specifying remote port numbers results in no records being collected (except for local-to-local connections).

The JOBNAME or ASID keywords in the TRACE command response limits tracing to only activity related to the listed applications. Wildcards are not used, the full JOBNAME must be specified.  For TN3270 trace, JOBNAME specifies an LUNAME for the relevant session (do not use ASID).

 
 


F. Collecting an OSA Network (SYSTCPOT) Trace
See TechNote 1232599 for the OSA models and z/OS releases that support this function, and the general directions for using an external writer or for internal buffers. Do not specify any OPTIONS, JOBNAME, or ASID values on the reply to the TRACE command for the SYSTCPOT component.

For step 3, you need to start the OSAENTA processing in TCPIP. Repeat for each OSA port if multiple OSAs are to be traced.
VARY TCPIP,tcpipname,OSAENTA,PORTNAME=osaport,ON,CLEARFILTER,NOFILTER=ALL

The previous command causes all IP traffic to be traced. Add specification of one or more filters to reduce the volume of data recorded

There are many filters that can be specified on the OSAENTA command, the complete list can be seen in the IP System Administrator's Commands manual. Typically include the CLEARFILTER keyword is included to remove any previous settings when you start a new trace, otherwise any specifications are ANDed with any previous filters (that is, more specific criteria).

The default action when no filters are specified is to not record anything, except for discarded packets. If you do not want any filtering to be performed, specify NOFILTER=ALL.

The most common filters to use are:
  • An IP address (IPADDR=aa.bb.cc.dd) or a masked address (IPADDR=aa.bb.cc.00/24 or IPADDR=aa.bb.cc.dd,SUBNET=255.255.255.0). This address is typically for the remote system being traced.
  • A port number (PORTNUM=nnn). This value is typically for the server port, either local or remote.
  • Abbreviated tracing (ABBREV or ABBREV=len), which reduces the amount of data recorded per frame. Most current OSA models limit the amount of data to 224 bytes as a maximum (even if FULL is specified).
  • By default, frames discarded by the OSA are only traced if it is an unusual condition. ARP requests for IP addresses not owned by this OSA or frames with other MAC or IP addresses that are discarded are not traced by default. Specify the DISCARD=ALL option to see all discarded frames, or DISCARD=NONE to not see any discarded frames.
Fort step 6, use the following commands to clear out any prior filter specifications and close the OSAENTA device:

VARY TCPIP,tcpipproc,OSAENTA,PORTNAME=osaport,OFF
VARY TCPIP,tcpipproc,OSAENTA,PORTNAME=osaport,DEL

 
 

[{"Product":{"code":"SSSN3L","label":"z\/OS Communications Server"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":"All","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"1.6;1.7;1.8;1.9;1.10;1.11;1.12;1.13;2.1;2.2;2.3","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}}]

Document Information

Modified date:
16 June 2020

UID

swg21292013

Page Feedback