Traces

The traces pages display distributed tracing information available on a per-trace basis.

Traces page

You can search and filter the listed traces to track down traces of interest, and click any trace to see detailed information and visual displays of the trace across the integration capabilities.

Figure 1. Traces

Traces

Trace page

The Trace page provides detailed information about a single trace, including visual displays of the trace across the integration capabilities and all the spans that belong to it. For more information about the meaning of traces and spans, see overview page.

Figure 2. Trace details - flow

Trace details - flow

Figure 3. Trace details - spans

Trace details - flow

Trace summary

The top part of the page shows a summary of the trace. To the left, the Trace info panel shows the total trace duration in ms., the integration capabilities that were involved and the top services. To the right, the Trace overview panel shows a compact visual display of all spans, including the ability to zoom into a specific time range in order to display that time range in the following spans table.

Flow diagram

The flow diagram shows all the services, containers, and operations that were involved in the trace.

This diagram includes the following functionality:

  • Each node (circle) represents a runtime or operation in the trace, such as an API deployed on a specific gateway, a message flow deployed on a specific integration server, or a queue that belongs to a specific queue manager.

  • Each edge (arrow) represents a flow of data between two nodes, such as an HTTP request made from an API to a message flow, or an MQ message read from a queue by a message flow.

  • The border color of each node matches the integration capability color used throughout the Web Console (for example, App Connect spans are green).

  • Hover over a node or an edge to see a summary panel of its data.
    Figure 4. Node summary
    Node summary

  • Use the animation buttons on the left side of the toolbar to watch the sequence of the trace, step-by-step. During the animation, the Trace overview panel visualizes the elapsed time of the current step.

  • Use the buttons on the right side of the toolbar to change the zoom level and expand or collapse the size of the flow diagram.

  • Click a single node to display its data and the list of spans that belong to this node on the right panel.
    Figure 5. Node summary
    Node right panel

  • Click a single edge to display its data on the right panel.
    Figure 6. Node summary
    Edge right panel

  • Click a span on the right panel to see the span's full details panel.
    Click on the left or the right border of that panel to toggle between a wide display and a narrow display.
    Click on the pin button to open several panels for side-by-side comparison of different spans.
    Figure 7. Span details
    Flow span details

  • The node summary panel and the right panel include links for accessing the logs and monitoring metrics of the relevant containers (pods) and the time frame for further investigation of errors and latency issues.
    Click Logs to access the logs of the container that sent the tracing data within the relevant time frame (uses Kibana by default).
    Click Monitoring to access the monitoring metrics of the pod that sent the tracing data within the relevant time frame (uses Grafana by default).
    For customizing and disabling these links, see Display system parameters.

  • Grab and drag the background to pan the diagram.

  • Press Shift and click on a node or an edge to add it to the selection.

  • Press Shift and draw a box around nodes or edges to add them to the selection.

  • Grab and drag a node to move it (and all other selected nodes) to a new location.

Spans table

The spans table shows all the spans that belong to the trace, ordered by hierarchy.

This table includes the following functionality:

  • The width of each bar represents the duration of the span.

  • The color of each bar matches the integration capability color used throught the Web Console (for example, MQ spans are purple).

  • The Level column displays the call depth of each span; level number 1 means that the span is the root span, level number 2 means that the span is a child of (or invoked by) the span at level 1, and so on. Click the level number next to the span to expand or collapse its children. Spans without a caret next to the level number do not have child spans.

  • The Level select allows to display only spans below a certain call depth (spans with a higher level are collapsed). Figure 8. Spans level select Spans level select

  • Hover over a span to see a summary panel of the span data. Figure 9. Span summary Span summary

  • Click on a span to see the span full details panel. Click on the left or the right border of that panel to toggle between a wide display and a narrow display. Click on the pin button to allow opening several panels for side-by-side comparison of different spans. Figure 10. Span details Span details

  • The span summary panel includes links for accessing the logs and monitoring metrics of the relevant containers or pods and the time for further investigation of errors and latency issues. Click Logs to access the logs of the container that sent that span within the relevant time frame (uses Kibana by default). Click Monitoring to access the monitoring metrics of the pod that sent that span within the relevant time frame (uses Grafana by default).
    For customizing and disabling these links, see Display system parameters.