Collecting trace for deployed flows

You can collect user and service trace to troubleshoot issues with deployed flows in App Connect.

About this task

If you can't get enough information from the logs about an issue in a deployed flow, trace can provide more detail about what is happening when code runs. When you trace an integration flow, you can see the path that messages take through your flow. Two types of trace are available in App Connect: user trace and service trace.
User trace
Enable user trace to debug your deployed solutions. User trace is less verbose than service trace.
Service trace
Service trace provides more comprehensive (verbose) tracing. Because service trace affects performance, enable it only if indicated by an error message or when you're directed to do so by IBM Support.
You can run one type of trace at a time. For example, if you start user trace and need to run service trace, you must stop user trace before you start service trace.
Warning: When you start tracing, you cause extra processing for every activity that you're tracing. Large quantities of data are generated, particularly for service trace, and you can expect performance to be affected while trace is active. You can limit this extra processing by restricting the length of time that you run trace.

Trace is not enabled by default. You can download trace at any time; you don't need to stop tracing first. Trace information is downloaded as a .zip file. A formatted log file contains a sequence of timestamped messages that record the activity in the deployed flow. When you reset trace, any user trace data that is already written to trace log files is deleted. You can reset trace for a running trace, and tracing resumes automatically after you reset it. You can start, stop, download, and reset trace for deployed flows that you created in both the IBM® App Connect Enterprise Toolkit and App Connect Designer.

You can collect trace for a deployed flow in two ways.

Starting trace for a deployed integration runtime

You can start, stop, reset, and download user and service trace from the menu of an integration runtime tile. The runtime tile indicates when trace is enabled.

About this task

Use this method to start trace for an integration runtime that is already started and contains a running integration. You might want to start trace to diagnose issues when you start to see errors in your logs. With this method, you can start and stop trace quickly to limit extra processing.

Procedure

The following steps describe how to collect user and service trace for a deployed integration runtime.

  1. From the Runtimes tab of the Manage page The icon that represents the Manage page, locate the tile for the relevant integration runtime.
    The integration runtime must be in a Ready state.
  2. Open the options menu on the tile. If trace isn't yet enabled, you can see the Start service trace and Start user trace options.
  3. To start trace, click the Start option for the trace that you want to enable.
    The trace options in the menu of the integration runtime tile include start, reset, and download service or user trace.
    A confirmation message is shown (Success: Started user trace) and the status of the integration runtime is updated to show that tracing is enabled.
    Screenshot that shows that the status on the integration runtime tile has changed from Ready to Ready (trace enabled)

    In the options menu for the integration runtime, the Start option for the trace that you enabled (for example, user trace) is replaced by a Stop option. The Start option for the alternative trace type (for example, service trace) is no longer visible because only one trace can be enabled per integration runtime. To start the alternative trace, stop the active trace first.

  4. To download trace log files, open the tile menu Icon that represents the options for the integration runtime and click the Download option for the appropriate type of trace. You can download log files when trace is running or after it is stopped.
    Trace log files are written to text files, which are downloaded in a .zip file. Depending on your browser, the file might be downloaded automatically to a configured download location or you might be prompted to choose a location. By default, the file is named instanceName-runtimeName-traceType-identifier.zip; for example, ac0abcd1efg-my-server-user-1677008332866.zip or ac0abcd1efg-my-server-service-1688119443977.zip.
    User trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains a .zip file that is called user-trace-files.zip, which contains one or more text files with the log entries. The name of the text files follows the format integration_server.runtimeName.userTrace.n.txt. The value of n is an integer between 0 and 4 because trace can be written to five trace log files in a rotational manner. When user trace is started, trace information is written to file 0. When that file is full, trace information is then written to the file 1. The process continues until file 4 is full, at which point the existing trace information in file 0 is overwritten with new information. The process continues in this rotational manner until user trace is stopped.
    Service trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains two .zip files that are called pod-logs.zip and service-trace-files.zip. The pod-logs.zip file contains logs from all containers in the pod. The service-trace-files.zip file contains the following text files.
    • A list of environment settings (env.txt).
    • An exception log (integration_server.runtimeName.exceptionLog.txt).
    • One or more trace log files (integration_server.runtimeName.trace.n.txt). The value of n is an integer between 0 and 3 because a large trace log can be divided into four files.
  5. To reset trace, open the tile menu and click the Reset option for the appropriate type of trace.
    A confirmation message is shown (Success: Reset user trace) and all the trace information that is written to trace log files for that integration runtime is erased. If you reset trace while trace is enabled, tracing resumes automatically.
  6. To stop trace, open the tile menu and click the Stop option for the appropriate type of trace.
    You can see a Stop option only when trace is enabled. A confirmation message is shown (Success: Stopped user trace) and the status of the tile is updated to show that trace is not enabled.

Starting trace when the integration runtime starts

If you want to start trace when an integration runtime starts, you can create the integration runtime with a server.conf.yaml configuration, which defines user and service trace settings.

About this task

When you deploy an integration to an integration runtime, you can configure that integration runtime by using a server.conf.yaml file. This file contains properties to start user or service trace when the integration runtime starts so that you can collect trace when the integration first runs. To stop trace for the integration runtime, you must update the server.conf.yaml configuration to disable the trace option. The integration runtime is restarted automatically when you update the configuration.

Procedure

To start user or service trace when an integration runtime starts, complete the following steps.

  1. Get a copy of a server.conf.yaml file from the local directory where IBM App Connect Enterprise Toolkit is installed.
    For example, on Windows, C:\Program Files\IBM\ACE\13.0.1.0\server\sample\configuration. A sample file is provided in the attached archive for reference: ACEaaSserverconfyaml/server.conf.zip. For more information, see Configuring an integration server by modifying the server.conf.yaml file in the IBM App Connect Enterprise documentation.
  2. To enable trace in the server.conf.yaml file, set the trace or userTrace properties, then save the file.
    The following example shows the trace and userTrace properties in a server.conf.yaml file in which user trace is enabled with a debug value.
    ...
#trace: 'none'               # choose 1 of : none|debug|debugTree|diagnostic|diagnosticTree
#traceSize: '1G'             # Set the service trace size

#traceNodeLevel: true        # Enable or disable message flow 'Trace nodes'

userTrace: 'debug'           # choose 1 of : none|debug|debugTree
userTraceSize: '1G'         # Set the user trace size
...
  3. On the Integrations tab of the Manage page, click Deploy integration, select the integration that you want to deploy, and click Next.
  4. Choose an existing runtime or create one.
    For more information, see Deploying integrations.
  5. On the Add configurations page, click Create configuration.
  6. In Type, select server.conf.yaml and give the configuration an appropriate name and description so that you can recognize it when you need to update it.
  7. Either import your server.conf.yaml file by clicking Import serverconf file The icon to import a serverconf file, or paste the contents of your file.
    Screenshot shows the Create configuration panel. The Type is set to server.conf.yaml, the name is start-trace, and the contents of a server.conf.yaml file are shown with user trace set to debug.
  8. Click Create, and make sure that the configuration is selected so that it can be applied to your integration.
    You can select just one server.conf.yaml file to apply to the integration. Configurations are listed alphabetically and you might need to go to another page to find your configuration.
  9. Click Next, review the summary, then click Deploy.

    On the Runtimes tab, a tile represents your new integration runtime. Initially, the status is Updating. Refresh the page to update the status. When the integration runtime starts, the status changes to Ready (trace enabled).

  10. To download trace log files, open the tile menu Icon that represents the options for the integration runtime and click the Download option for the appropriate type of trace. You can download log files when trace is running or after it is stopped.
    Trace log files are written to text files, which are downloaded in a .zip file. Depending on your browser, the file might be downloaded automatically to a configured download location or you might be prompted to choose a location. By default, the file is named instanceName-runtimeName-traceType-identifier.zip; for example, ac0abcd1efg-my-server-user-1677008332866.zip or ac0abcd1efg-my-server-service-1688119443977.zip.
    User trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains a .zip file that is called user-trace-files.zip, which contains one or more text files with the log entries. The name of the text files follows the format integration_server.runtimeName.userTrace.n.txt. The value of n is an integer between 0 and 4 because trace can be written to five trace log files in a rotational manner. When user trace is started, trace information is written to file 0. When that file is full, trace information is then written to the file 1. The process continues until file 4 is full, at which point the existing trace information in file 0 is overwritten with new information. The process continues in this rotational manner until user trace is stopped.
    Service trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains two .zip files that are called pod-logs.zip and user-trace-files.zip. The pod-logs.zip file contains logs from all containers in the pod. The user-trace-files.zip file contains the following text files.
    • A list of environment settings (env.txt).
    • An exception log (integration_server.runtimeName.exceptionLog.txt).
    • One or more trace log files (integration_server.runtimeName.trace.n.txt). The value of n is an integer between 0 and 3 because a large trace log can be divided into four files.
  11. To stop trace for the integration runtime, update the server.conf.yaml configuration.
    1. Go to the Configurations tab of the Manage page.
    2. Find the server.conf.yaml configuration that is applied to the integration runtime, and from the action menu Icon that represents the action menu, click Update.
    3. Disable the trace options by commenting them out.
      For example, #userTrace: 'debug'.
    4. Click Update.
      You're prompted to confirm the update and reminded that the update restarts the associated integration runtime. When you click Confirm, the integration runtime is restarted with trace disabled.