Running a trace for uDAPL over InfiniBand connections

When reporting a problem to IBM Service, a detailed trace of the uDAPL communication layer and lower communication layers might be requested. These instructions can be used as a general reference for taking AIX® traces.

About this task

Important: Starting from version 11.5.5, support for Infiniband (IB) adapters as the high-speed communication network between members and CFs in Db2® pureScale® on all supported platforms is deprecated and will be removed in a future release. Use Remote Direct Memory Access over Converged Ethernet (RoCE) network as the replacement.

Enabling the AIX system trace facility (OFF by default) might impact your system's performance depending on how it is used. As a result, use the trace facility to diagnose connectivity problems only when directed by IBM Service. The optimal values that are to be used for the trace mode, trace buffer size, file size, and trace hooks vary depending on the type of problem. IBM Service analyzes the problem and gives recommended values for these variables.

Procedure

To trace uDAPL over InfiniBand connections:

  1. Adjust any component trace levels to refine which events should be traced for certain trace hooks.
    For example, to include detail level events for the InfiniBand communication layer in any subsequent AIX system trace, run the following command:
    ctctrl -t network_ib -r systracedetail
  2. Turn the trace on. To turn on the trace, use the trace command: 
    trace -T buffersize -L filesize -a -j trace_hooks
    where:
    • The default trace mode is used. Two global buffers are used to continuously gather trace data, with one buffer being written to the log file, while data is gathered in the other buffer
    • -T buffersize specifies the trace buffer size
    • -L filesize specifies the output trace log file size
    • -a specifies to run the trace in the background
    • -j trace_hooks specifies the type of event to trace. Multiple events can be specified as a comma-separated list
    Note: Values for trace mode, buffer scope, buffersize, filesize, and trace_hooks depend on the problem that is being experienced. Contact IBM Service for recommended values.
  3. Reproduce the problem
  4. Turn off the trace. Use the trcstop command.
  5. Dump the trace buffer to a file. Use the trcrpt command: 
    trcrpt > filename

    where filename specifies the file you dump the trace buffer to. For information about the AIX tracing facility, seeAIX Information Center: Trace Facility

    AIX also supports in-memory only traces for some components. These include default on traces, that were selected such that there is minimal performance impact. The traces are not written to a disk log file without explicit action by IBM Service personnel. To increase the InfiniBand in-memory tracing to a detailed level, use the following command:
    ctctrl -t network_ib -r memtracedetail