Investigating AI interactions

To investigate and understand AI interactions, use tracing to capture execution details and review how the AI agent processed a request. Tracing helps you troubleshoot unexpected behavior, review tool and model activity, and analyze execution timelines.

Use the Trace Viewer to visualize the execution and inspect conversation turns, execution steps, tool invocations, and performance metrics.

Enabling and disabling tracing

Tracing captures detailed execution information for all agentic AI requests while it is enabled.

To start collecting trace data, from the AI chat window, switch the Tracing toggle to On. Then, reproduce the scenario that you want to analyze.

Tracing remains active for a maximum of 10 minutes. After 10 minutes, tracing automatically stops to minimize performance impact. You can also stop tracing by manually switching the Tracing toggle to Off.

When tracing collection stops, the system automatically downloads the collected trace data as a usage.json file. You can import this file into the Trace Viewer to analyze the execution details and identify potential issues.

Trace file contents

The downloaded usage.json file contains diagnostic information about the traced agentic AI requests. Depending on the execution, the file contains details such as these:

  • Request and execution metadata.
  • Conversation turns.
  • Model invocations.
  • Tool invocations and execution status.
  • Execution timelines.
  • Token usage.
  • Latency and performance metrics.
  • Execution flow information.
  • Error details.

You can review this information in the Trace Viewer or share the file with development and support teams when you investigate issues.

Importing trace files into the Trace Viewer

Import the downloaded usage.json file into the Trace Viewer to visualize and analyze the captured data. The Trace Viewer provides an interactive visualization of the data that is captured in the usage.json file.

To import a trace file into the Trace Viewer, complete the following steps from the Sterling Order Management System developer toolkit (DTK).

  1. In the chat application, from the navigation side panel, select Trace Viewer.
  2. Click Import JSON and select the downloaded usage.json file.

The trace file is loaded and displayed in the Trace Viewer for analysis.

Reviewing captured data

The Trace Viewer provides a visual representation of how requests are processed and shows how agents, models, and tools process requests. Review the trace data in the Trace Viewer to understand request processing and identify potential issues. Trace data helps you in the following ways.

  • Understand how an AI agent processed a request.
  • Verify that the expected models and tools were used.
  • Review the sequence of tool invocations.
  • Identify failed or timed-out operations.
  • Analyze execution timelines and performance metrics.
  • Compare successful and unsuccessful request flows.
  • Share trace information with development or support teams when you investigate issues.
The Trace Viewer includes several sections.
Conversation turns
The Conversation turns section displays the user interactions that were captured during the tracing session. Select a conversation turn to view the associated execution details.
Turn summary
The Turn summary section provides information about the selected conversation turn, including the following data:
  • Execution status
  • User message
  • Session ID
  • Request ID
  • Number of execution steps
  • Interruption status
Execution timeline
The Execution timeline section displays the sequence of execution steps for a conversation turn, including execution status and duration. Expand individual steps to view details such as agent activity, model invocations, tool calls, and errors.
Execution flow
The Execution flow section displays a visual representation of how the request moved through agents and tools. Use this view, and expand or collapse nodes, to understand the execution path and inspect agent activity at a more granular level.

Best practices

  • Enable tracing immediately before you investigate an issue.
  • Perform only the actions that are required to reproduce the issue.
  • Turn tracing off after you reproduce the issue.
  • Review the captured trace in the Trace Viewer.
  • Attach the usage.json file when you report issues.
  • Do not modify the usage.json file before you share it.