Collecting trace for deployed flows

You can collect user and service trace to troubleshoot issues with deployed flows in IBM® App Connect Enterprise as a Service.

About this task

If you can't get enough information from the logs about an issue in a deployed flow, trace can provide more detail about what is happening when code runs. When you trace an integration flow, you can see the path that messages take through your flow. Two types of trace are available in App Connect Enterprise as a Service: user trace and service trace.
User trace
Enable user trace to debug your deployed solutions. User trace is less verbose than service trace.
Service trace
Service trace provides more comprehensive (verbose) tracing. Because service trace affects performance, enable it only if indicated by an error message or when you're directed to do so by IBM Support.
You can run one type of trace at a time. For example, if you start user trace and need to run service trace, you must stop user trace before you start service trace.
Warning: When you start tracing, you cause extra processing for every activity that you're tracing. Large quantities of data are generated, particularly for service trace, and you can expect performance to be affected while trace is active. You can limit this extra processing by restricting the length of time that you run trace.

Trace is not enabled by default. You can download trace at any time; you don't need to stop tracing first. Trace information is downloaded as a .zip file. A formatted log file contains a sequence of timestamped messages that record the activity in the deployed flow. When you reset trace, any user trace data that is already written to trace log files is deleted. You can reset trace for a running trace, and tracing resumes automatically after you reset it. You can start, stop, download, and reset trace for deployed flows that you created in both the IBM App Connect Enterprise Toolkit and App Connect Designer.

If you're on the VPC hours plan, you can collect trace for a deployed flow in two ways.

If you're on the flow runs plan, you don't create an integration runtime to deploy your flows. You can collect trace by completing the steps in Starting trace for a deployed flow.

Starting trace for a deployed integration runtime

You can start, stop, reset, and download user and service trace from the menu of an integration runtime tile in the App Connect Dashboard. The runtime tile shows when trace is enabled.

About this task

If you're on the VPC hours plan, use this method to start trace for an integration runtime that is already started and contains a running integration. You might want to start trace to diagnose issues when you start to see errors in your logs. With this method, you can start and stop trace quickly to limit extra processing.

Procedure

The following steps describe how to collect user and service trace for a deployed integration runtime.

  1. From the Runtimes page of the App Connect Dashboard, locate the tile for the relevant integration runtime.
    The integration runtime must be in a Ready state.
  2. Open the options menu on the tile. If trace isn't yet enabled, you can see the Start service trace and Start user trace options.
  3. To start trace, click the Start option for the trace that you want to enable.
    Screenshot that shows the trace options in the menu of the integration runtime tile in the App Connect Dashboard
    A confirmation message is shown (Success: Started user trace) and the status of the integration runtime is updated to show that tracing is enabled.
    Screenshot that shows that the status on the integration runtime tile has changed from Ready to Ready (trace enabled)

    In the options menu for the integration runtime, the Start option for the trace that you enabled (for example, user trace) is replaced by a Stop option. The Start option for the alternative trace type (for example, service trace) is no longer visible because only one trace can be enabled per integration runtime. To start the alternative trace, stop the active trace first.

  4. To download trace log files, open the tile menu Icon that represents the options for the integration runtime and click the Download option for the appropriate type of trace. You can download log files when trace is running or after it is stopped.
    Trace log files are written to text files, which are downloaded in a .zip file. Depending on your browser, the file might be downloaded automatically to a configured download location or you might be prompted to choose a location. By default, the file is named instanceName-runtimeName-traceType-identifier.zip; for example, ac0abcd1efg-my-server-user-1677008332866.zip or ac0abcd1efg-my-server-service-1688119443977.zip.
    User trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains a .zip file that is called user-trace-files.zip, which contains one or more text files with the log entries. The name of the text files follows the format integration_server.runtimeName.userTrace.n.txt. The value of n is an integer between 0 and 4 because trace can be written to five trace log files in a rotational manner. When user trace is started, trace information is written to file 0. When that file is full, trace information is then written to the file 1. The process continues until file 4 is full, at which point the existing trace information in file 0 is overwritten with new information. The process continues in this rotational manner until user trace is stopped.
    Service trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains two .zip files that are called pod-logs.zip and service-trace-files.zip. The pod-logs.zip file contains logs from all containers in the pod. The service-trace-files.zip file contains the following text files.
    • A list of environment settings (env.txt).
    • An exception log (integration_server.runtimeName.exceptionLog.txt).
    • One or more trace log files (integration_server.runtimeName.trace.n.txt). The value of n is an integer between 0 and 3 because a large trace log can be divided into four files.
  5. To reset trace, open the tile menu and click the Reset option for the appropriate type of trace.
    A confirmation message is shown (Success: Reset user trace) and all the trace information that is written to trace log files for that integration runtime is erased. If you reset trace while trace is enabled, tracing resumes automatically.
  6. To stop trace, open the tile menu and click the Stop option for the appropriate type of trace.
    You can see a Stop option only when trace is enabled. A confirmation message is shown (Success: Stopped user trace) and the status of the tile is updated to show that trace is not enabled.

Starting trace when the integration runtime starts

If you want to start trace when an integration runtime starts, you can create the integration runtime with a server.conf.yaml configuration, which defines user and service trace settings.

About this task

If you're on the VPC hours plan, when you deploy an integration to an integration runtime, you can configure that integration runtime by using a server.conf.yaml file. This file contains properties to start user or service trace when the integration runtime starts so that you can collect trace when the integration first runs. To stop trace for the integration runtime, you must update the server.conf.yaml configuration to disable the trace option. The integration runtime is restarted automatically when you update the configuration.

Procedure

To start user or service trace when an integration runtime starts, complete the following steps.

  1. Get a copy of a server.conf.yaml file from the local directory where IBM App Connect Enterprise Toolkit is installed.
    For example, on Windows, C:\Program Files\IBM\ACE\12.0.8.0\server\sample\configuration. A sample file is provided in the attached archive for reference: ACEaaSserverconfyaml/server.conf.zip. For more information, see Configuring an integration server by modifying the server.conf.yaml file in the IBM App Connect Enterprise documentation.
  2. To enable trace in the server.conf.yaml file, set the trace or userTrace properties, then save the file.
    The following example shows the trace and userTrace properties in a server.conf.yaml file in which user trace is enabled with a debug value.
    ...
#trace: 'none'               # choose 1 of : none|debug|debugTree|diagnostic|diagnosticTree
#traceSize: '1G'             # Set the service trace size

#traceNodeLevel: true        # Enable or disable message flow 'Trace nodes'

userTrace: 'debug'           # choose 1 of : none|debug|debugTree
userTraceSize: '1G'         # Set the user trace size
...
  3. Go to the App Connect Dashboard, click Deploy integrations, choose an integration size, and select the BAR file that contains your integration.
    For more information, see Deploying integrations on the VPC plan.
  4. On the Configuration page, click Create configuration.
  5. In Type, select server.conf.yaml and give the configuration an appropriate name and description so that you can recognize it when you need to update it.
  6. Either import your server.conf.yaml file by clicking Import serverconf file The icon to import a serverconf file, or paste the contents of your file.
    Screenshot shows the Create configuration panel. The Type is set to server.conf.yaml, the name is start-trace, and the contents of a server.conf.yaml file are shown with user trace set to debug.
  7. Click Create, and make sure that the configuration is selected so that it can be applied to your integration.
    You can select just one server.conf.yaml file to apply to the integration.
  8. Complete the creation of the integration runtime by naming it and setting any other necessary properties.

    You are returned to the Runtimes page, where you can see a tile that represents your new integration runtime. Initially, the status is "pending". Refresh the page to update the status. When the integration runtime starts, the status changes to Ready (trace enabled).

  9. To download trace log files, open the tile menu Icon that represents the options for the integration runtime and click the Download option for the appropriate type of trace. You can download log files when trace is running or after it is stopped.
    Trace log files are written to text files, which are downloaded in a .zip file. Depending on your browser, the file might be downloaded automatically to a configured download location or you might be prompted to choose a location. By default, the file is named instanceName-runtimeName-traceType-identifier.zip; for example, ac0abcd1efg-my-server-user-1677008332866.zip or ac0abcd1efg-my-server-service-1688119443977.zip.
    User trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains a .zip file that is called user-trace-files.zip, which contains one or more text files with the log entries. The name of the text files follows the format integration_server.runtimeName.userTrace.n.txt. The value of n is an integer between 0 and 4 because trace can be written to five trace log files in a rotational manner. When user trace is started, trace information is written to file 0. When that file is full, trace information is then written to the file 1. The process continues until file 4 is full, at which point the existing trace information in file 0 is overwritten with new information. The process continues in this rotational manner until user trace is stopped.
    Service trace files
    The downloaded .zip file contains a directory that is named after your runtime, such as my-server-ir-64f9b677d-7jpv5. If your runtime has multiple replicas, a directory is created for each replica. Each directory contains two .zip files that are called pod-logs.zip and user-trace-files.zip. The pod-logs.zip file contains logs from all containers in the pod. The user-trace-files.zip file contains the following text files.
    • A list of environment settings (env.txt).
    • An exception log (integration_server.runtimeName.exceptionLog.txt).
    • One or more trace log files (integration_server.runtimeName.trace.n.txt). The value of n is an integer between 0 and 3 because a large trace log can be divided into four files.
  10. To stop trace for the integration runtime, update the server.conf.yaml configuration.
    1. Go to the Configuration page Icon for the Configuration page.
    2. Find the server.conf.yaml configuration that is applied to the integration runtime, and from the action menu Icon that represents the action menu, click Update.
    3. Disable the trace options by commenting them out.
      For example, #userTrace: 'debug'.
    4. Click Update.
      You're prompted to confirm the update and reminded that the update restarts the associated integration runtime. When you click Confirm, the integration runtime is restarted with trace disabled.

Starting trace for a deployed flow

You can start, stop, reset, and download user and service trace from the menu of a deployed flow on the Manage tab. The tile shows when trace is enabled.

About this task

If you're on the flow runs plan, you can use this method to start trace for a flow or BAR file that is deployed and active. You might want to start trace to diagnose issues when you start to see errors in your logs. With this method, you can start and stop trace quickly to limit extra processing.
Warning: When you start, stop, reset, or collect user trace for a deployed Designer flow, user tracing is affected for all deployed Designer flows.

Procedure

The following steps describe how to collect user and service trace for a deployed flow or BAR file.

  1. From the Manage tab, locate the tile for the relevant flow or BAR file.
    The flow or BAR file must be in an Active state.
  2. Open the options menu on the tile Icon that represents the options for the integration runtime. If trace isn't yet enabled, you can see the Start service trace and Start user trace options.
  3. To start trace, click the Start option for the trace that you want to enable.
    Screenshot that shows the trace options in the menu of the deployed flow tile on the Manage tab.

    A confirmation message is shown (Success: Started user trace) and in the options menu for the flow or BAR file, the Start option for the trace that you enabled (for example, user trace) is replaced by a Stop option. The Start option for the alternative trace type (for example, service trace) is no longer visible because only one trace can be enabled per integration runtime. To start the alternative trace, stop the active trace first.

  4. To download trace log files, open the tile menu and click the Collect option for the appropriate type of trace. You can download log files when trace is running or after it is stopped.
    Trace log files are written to text files, which are downloaded in a .zip file. Depending on your browser, the file might be downloaded automatically to a configured download location or you might be prompted to choose a location. By default, the file is named instanceName-deploymentName-traceType-identifier.zip; for example, ac0abcd1efg-myflow-user-1677008332866.zip or ac0abcd1efg-myflow-service-1688119443977.zip.
    User trace files
    The downloaded .zip file contains a directory that is named after your deployment, such as myflow-ir-64f9b677d-7jpv5. The directory contains a .zip file that is called user-trace-files.zip, which contains one or more text files with the log entries. The name of the text files follows the format integration_server.deploymentName.userTrace.n.txt. The value of n is an integer between 0 and 4 because trace can be written to five trace log files in a rotational manner. When user trace is started, trace information is written to file 0. When that file is full, trace information is then written to the file 1. The process continues until file 4 is full, at which point the existing trace information in file 0 is overwritten with new information. The process continues in this rotational manner until user trace is stopped.
    Service trace files
    The downloaded .zip file contains a directory that is named after your deployment, such as myflow-ir-64f9b677d-7jpv5. The directory contains two .zip files that are called pod-logs.zip and service-trace-files.zip. The pod-logs.zip file contains logs from all containers in the pod. The service-trace-files.zip file contains the following text files.
    • A list of environment settings (env.txt).
    • An exception log (integration_server.deploymentName.exceptionLog.txt).
    • One or more trace log files (integration_server.deploymentName.trace.n.txt). The value of n is an integer between 0 and 3 because a large trace log can be divided into four files.
  5. To reset trace, open the tile menu and click the Reset option for the appropriate type of trace.
    A confirmation message is shown (Success: Reset user trace) and all the trace information that is written to trace log files for that integration runtime is erased. If you reset trace while trace is enabled, tracing resumes automatically.
  6. To stop trace, open the tile menu and click the Stop option for the appropriate type of trace.
    You can see a Stop option only when trace is enabled. A confirmation message is shown (Success: Stopped user trace).