Tracing context in log files

IBM® Cloud Pak for Network Automation generates logs across all of its microservices and provides a tracing context so that you can isolate and track the log entries for a specific intent request.

Tracing context

All IBM Cloud Pak for Network Automation actions have an associated tracing context, which is a collection of identifiers that can be used to collect log entries that relate to a specific intent request. The tracing context enables customers to categorize and track the progress of an API request through the system, and is useful for troubleshooting and monitoring performance. For example, if you have multiple applications that are making REST requests to IBM Cloud Pak for Network Automation, you can add tracing information that shows which of your applications made a particular request.

The primary element of the tracing context is the transaction ID, which must be unique and is available for all API requests. If you do not supply a transaction ID header in the original REST API request, then the point of ingress, which is normally the API gateway, randomly generates a unique transaction ID. The transaction ID is available for the rest of the transaction so that you can track an individual transaction throughout the system.

Table 1. API transaction ID header
Key	Value
x-tracectx-transactionId	`<any_unique_id>`

The tracing context, containing the transaction ID and any other context information from the HTTP headers, is passed on in calls to other microservices and is included in any log messages that are related to the API request. The trace headers are available as indexed fields in the logs so that you can trace the end-to-end set of logs for a specific transaction and view the log entries chronologically.

Logs that are generated by the IBM Cloud Pak for Network Automation services and any services that it interacts with, such as resource drivers, are supplied with the set of tracing context headers so that end-to-end log isolation can be achieved in a highly concurrent system.

Adding headers to the tracing context

At the point of ingress to IBM Cloud Pak for Network Automation, you can provide a tracing context by adding x-tracectx-<name>:<value> headers to the original REST API request. The names are converted to lowercase characters in the log output, as the HTTP headers that are used to make the REST calls are case-sensitive.

For example, to add a transaction ID to your tracing context, add the following header, with a sample ID value, to the REST request:

"x-tracectx-transactionid": "1234567-abcd-def0-1234-56789abcdef0"

The following information is added to all the log messages that are written by microservices that handle the request:

"x-tracectx.transactionid":"1234567-abcd-def0-1234-56789abcdef0"

You can also add other tracing headers to your REST requests. Any header that uses the "x-tracectx-<name>": "<value> format is added to the logs.

For example, if you have multiple applications that are making REST requests, you can include a request header that shows which of your applications made a particular request. You can add the following header, which includes a sample application value, to the REST request:

"x-tracectx-application": "AcmeCSMF"

The following information is added to all the log messages that are written by microservices that handle the request:

"x-tracectx.application":"AcmeCSMF"

Tracing context in Cancel, Retry, and Rollback intent requests

The tracing context headers in Cancel, Retry, or Rollback intent requests (also known as intent amendment requests) might be different than the headers in the original request that the amendment relates to. The context headers that are used in the log messages for the amendment request depend on the type of context header and whether the amendment request is accepted or not. In some scenarios, the context headers from the original intent request are used in the log messages.

For example, your CreateAssembly request fails and you use the Retry request to create the assembly. The original CreateAssembly request might be received from the REST API and might contain tracing context headers. However, the Retry amendment request might be received from a different source, for example, the UI. You might not be able to include the same tracing context headers in the amendment request as the original request.

The following table shows the different types of context headers and whether the context headers from the amendment request or the original intent request are written to the logs:

Type of header	Log messages before amendment accepted	Log messages after amendment accepted
`x-tracectx-tag`	From amendment request	From amendment request
Other `x-tracectx-`	From amendment request	From original request

For x-tracectx-tag headers, the context headers from the amendment request are always used in the log messages for the amendment request. These headers are primarily used to categorize metrics, for example, "x-tracectx-tag-source": "portal".

For other x-tracectx- headers, for example,

"x-tracectx-application":
"AcmeCSMF"

, the context headers that are used in the log messages for the amendment request depends on whether the request is accepted or not:

Before the amendment request is accepted, the context headers from the amendment request are used.
After the amendment request is accepted, the context headers from the original intent request are used. In the example, the context headers from the CreateAssembly request are used in the log messages for the Retry request. By using the headers from the original request, the correlation with the requesting system is retained.

Viewing trace information in the Kibana UI

If you deployed Elasticsearch, Fluentd, and Kibana (EFK) for application logging, you can use the Kibana UI to view and filter the logs that are generated by IBM Cloud Pak for Network Automation. The tracing context attributes, for example, transactionid, are configured as filterable fields for logs.

For example, on a highly concurrent system you might want to view all of the log messages that relate to a specific API request. To see the end-to-end set of logs for a specific transaction, you can add a filter in Kibana to view log messages for that transaction only. For example, you can add a filter for transaction ID 1234567-abcd-def0-1234-56789abcdef0.

Viewing trace information in Red Hat OpenShift

You can use the Red Hat® OpenShift® Container Platform console or the Red Hat OpenShift command-line interface (CLI) to view the logs that are generated by the Cloud Pak. To view log messages that contain specific tracing context values, you can use command-line text-matching tools, such as grep. For more information about viewing logs in OpenShift, see Opening logs in Red Hat OpenShift.