Diagnostic data

You might need to collect multiple pieces of data to accurately diagnose problems, such as the following:

Console messages for ADNR
syslogd log messages for ADNR (possibly including DEBUG - 64 level trace)
Name server log data for the name servers that are managed by ADNR
ADNR address space dump and snap output
SYSTCPIP CTRACE for the TCP/IP stack where ADNR and the GWM are running
Packet traces of GWM data
Netstat displays for the connection between ADNR and the GWM
A listing of the zone data from the managed name server or name servers
z/OS® Load Balancing Advisor log data and displays. See Diagnosing problems with the z/OS Load Balancing Advisor for information about the Load Balancing Advisor.

Note: syslogd is the only logging facility that ADNR uses. Useful diagnostic information might be lost if syslogd is not running before you run ADNR.

ADNR triggers address space dumps when certain unexpected error conditions are encountered while communicating with a GWM. Just after connecting to the GWM, ADNR enters a negotiation phase. During negotiation, a series of architected SASP requests are sent to the GWM; for each request, an architected SASP reply is received from the GWM. If the negotiation does not successfully complete, ADNR closes the connection, increases the logging level, establishes a new connection to the GWM, and retries the negotiation.

If the negotiation fails a second time, ADNR dumps its address space; the dump title header is ADNR Dump - Neg Failed and the logging level is restored to its original configured value and the connection is closed. Retries continue at 1 minute intervals using the configured logging level.

After completing the negotiation phase, the GWM might send an unsolicited SendWeights message to ADNR; the message contains information about the changed state of resources that the ADNR application registered to the GWM during negotiation. If the SendWeights message contains an architectural error, ADNR closes the connection, increases the logging level, establishes a new connection to the GWM, and completes negotiation with the GWM. If the next SendWeights message received from the GWM contains an error, ADNR dumps its address space; the dump title header is as follows:

ADNR Dump - Rcv Failed

The logging level is restored to its original configured value and the connection is closed. Retries continue at 1 minute intervals using the configured logging level.

These types of errors generally occur because incompatible levels of maintenance are applied to the GWM and ADNR. After being started, ADNR dumps its address space only once when these types of errors are detected. For further diagnosis, collect the ADNR log and address space dump. Review the log to determine the type of error that occurred. Review the PTF requirements of any recently installed PTFs. If you cannot correct the problem with additional maintenance or a configuration change, then contact IBM® Service.

For other types of errors, both a CEEDUMP and address space snap output might be produced and written to the data sets or files that are specified by the start procedure CEEDUMP and CEESNAP DD statements, respectively.

If ADNR abnormally terminates (for example an 0Cx abend occurs), then an unformatted SYSMDUMP is written to the data set that is specified by the start procedure SYSMDUMP DD statement. If you override the Language Environment® run-time option TERMTHDACT during the installation or start procedure, then the SYSMDUMP may not be produced, or a CEEDUMP may be produced instead. Therefore, you should not override the TERMTHDACT run-time option. See z/OS Language Environment Programming Guide for more information about Language Environment runtime options.

In other situations, the z/OS operator might need to dump the ADNR address space manually. See z/OS MVS Diagnosis: Tools and Service Aids for more information about obtaining a dump.

SYSTCPDA CTRACE (packet trace) data of the SASP protocol messages sent between ADNR and GWM may be needed. See TCP/IP services traces and IPCS support for details about how to use the IP packet trace facility.

Restriction: When encrypting data, the packet trace data will be encrypted. Use MESSAGE level log messages instead.

A SYSTCPIP component trace on the TCP/IP stacks used by ADNR and its associated GWM shows data that is flowing between them. Start the trace by specifying OPTIONS=(PFS,TCP,UDP,INTERNET),JOBNAME=(server) on both stacks, where the server value is the ADNR or GWM address space (or both names separated by a comma if they are using the same stack). See TCP/IP services traces and IPCS support for more information.

You can dump the contents of the DNS zones managed by ADNR by issuing the domain information grouper (dig) command from z/OS UNIX:

dig @server zone axfr -p port -k key_file 2>zone_xfer.err 1>zone_xfer.out

where

server: The server IP address or host name which contains the zone managed by ADNR
zone: The zone being managed by ADNR whose contents are being dumped
port: Optionally specify the port number which the DNS server is listening on for queries
key_file: Optionally specify the key file that is used to sign transactions for this zone
zone_xfer: Redirect the stdout and stderr file streams of the command to two distinct z/OS UNIX files.

See z/OS Communications Server: IP System Administrator's Commands for more information about the dig command.

The following Netstat command displays stacks that have affinity with ADNR:

ALLConn/-a, COnn/-c

This command is used to determine whether there is an active connection between ADNR and the GWM. ALLConn/-a displays information for all TCP connections and UDP sockets, including some recently closed ones. COnn/-c displays information about each active TCP connection and UDP socket. See z/OS Communications Server: IP System Administrator's Commands, Monitoring the TCP/IP network, Netstat section for information about using Netstat commands.