Configuring OpenTelemetry trace for an integration server

You can configure OpenTelemetry trace for all message flows in an integration server and export span data to an OpenTelemetry collector.

About this task

OpenTelemetry trace is supported for IBM® App Connect Enterprise integration servers on the Linux® x86-64, Linux zSeries, Linux on POWER® Systems - Little Endian, AIX®, and Windows platforms.

When OpenTelemetry is enabled for an integration server, spans are created for all message flow nodes that support OpenTelemetry, including callable flow nodes. For supported transport input nodes, spans are created and then run until the message flow transaction for the flow is completed or rolled back. For supported transport request nodes, spans are created and then run for the duration of the node interaction with the transport (for example, sending and receiving an HTTP message).

The following transport message flow nodes support OpenTelemetry trace:
  • MQ:
    • MQGet node
    • MQInput node
    • MQOutput node
    • MQPublication node
    • MQReply node
  • HTTP:
    • HTTPAsyncRequest node
    • HTTPAsyncResponse node
    • HTTPInput node
    • HTTPReply node
    • HTTPRequest node
  • REST:
    • RESTAsyncRequest node
    • RESTAsyncResponse node
    • RESTRequest node
  • SOAP:
    • SOAPAsyncRequest node
    • SOAPAsyncResponse node
    • SOAPInput node
    • SOAPReply node
    • SOAPRequest node

When OpenTelemetry trace is enabled, a message flow span is created for callable flows starting with a CallableInput node, and a span is created around the CallableFlowInvoke node. For asynchronous callable flow nodes, a message flow span is created for flows starting with a CallableFlowAsyncResponse node, and a span is created around the CallableFlowAsyncInvoke node.

The following Callable flow nodes support OpenTelemetry trace:
  • CallableInput
  • CallableReply
  • CallableFlowInvoke
  • CallableFlowAsyncInvoke
  • CallableFlowAsyncResponse

You can enable and configure OpenTelemetry trace by setting properties in the OpenTelemetry section of the server.conf.yaml file. By default, OpenTelemetry trace is not enabled.

If you want to send the OpenTelemetry trace to a secured observability backend, you can configure the integration server to specify an opentelemetry credential, by setting properties in the OpenTelemetry section of the server.conf.yaml file: 

#openTelemetryHttpSecurityId: '' # Specify an opentelemetry credential used in the HTTP exporter.
#openTelemetryGrpcSecurityId: '' # Specify an opentelemetry credential used in the GRPC exporter.

You set the openTelemetryHttpSecurityId and openTelemetryGrpcSecurityId properties to the name of a credential in the vault, with a credential-type of opentelemetry. The details that are retrieved from the referenced credential are then populated into the HTTP Authorization header. The value of the openTelemetryGrpcSecurityId property determines the credential that is used to configure the GRPC exporter, and the value of the openTelemetryHttpSecurityId property determines the credential that is used to configure the HTTP exporter.

The opentelemetry credential can have one of the following authentication types: basic, accessToken, or apiKey. The authentication type sets the schema in the HTTP Authorization header in the requests that are sent to the OpenTelemetry backend, in the form Authorization: <schema> credentials. The authentication types set the schema to either Basic, Bearer, or ApiKey, as shown in the following table:
Table 1. Credential authentication types and HTTP headers
Credential auth-type HTTP header
basic Authorization: Basic <username>:<password>
accessToken Authorization: Bearer <access-token>
apiKey Authorization: ApiKey <api-key>

For the Basic and ApiKey schema, the credentials are Base64 encoded.

For more information about using OpenTelemetry, see OpenTelemetry considerations and limitations and Turning off OpenTelemetry for a message flow.

Procedure

Configure OpenTelemetry trace for all flows on an integration server by completing the following steps:

  1. Open the server.conf.yaml configuration file for your integration server.

    You can edit the file by using the built-in YAML editor that is provided in the IBM App Connect Enterprise Toolkit, either by double-clicking the file in the Application Development view or by right-clicking the file and selecting Open with > YAML editor. If you choose to edit the file by using a plain text editor, ensure that you do not include any tab characters (which are not valid in YAML) and use a YAML validation tool to validate the contents of your file.

  2. In the ResourceManagers section of the server.conf.yaml file, set the properties that you require in the OpenTelemetryManager subsection:
    • Set the openTelemetryEnabled property to true to enable OpenTelemetry for all message flows on the integration server. By default, this property is set to false. For information about disabling OpenTelemetry for an individual message flow after it has been enabled on the integration server, see Turning off OpenTelemetry for a message flow.
    • Use the openTelemetryServiceName property to override the Service Name attribute for telemetry spans. By default, this property is set to App Connect Enterprise-<integration server name>.
    • Use the openTelemetryHostName property to override the Hostname attribute for telemetry spans.
    • Use the exporterOtlpGrpcEndpoint property to specify the GRPC endpoint to which the OpenTelemetry span data is to be exported. For example: 'telemetryagent.host.com:4317'.
    • Use the exporterOtlpHttpUrl property to specify the HTTP/JSON URL to which the OpenTelemetry span data is to be exported. For example, 'http://<hostname>:<port>/v1/traces'.
    • Use the openTelemetryHttpSecurityId property to specify the credential that is used to configure the HTTP exporter.
    • Use the openTelemetryGrpcSecurityId property to specify the credential that is used to configure the GRPC exporter.
    • Use the openTelemetryTruststoreType property to specify the type of the OpenTelemetry truststore. Valid values are PKCS12, PEM, and JKS. This property is used only for the GRPC exporter.
    • Use the openTelemetryTruststoreFile property to specify the location of the OpenTelemetry truststore. This property is used only for the GRPC exporter.
    • Use the openTelemetryTruststorePass property to specify the credential or resource alias containing the truststore password. This property is used only for the GRPC exporter.
    • Use the openTelemetryTrustAlias property to specify the alias of the trust certificate in the PKCS12 or JKS truststore.
    • Use the openTelemetrySpanProcessor property to specify the type of OpenTelemetry span processor to be used. You can specify either batch or simple. By default, this property is set to batch.
    • Use the openTelemetryBatchSpanOptions property to specify the OpenTelemetry batch span processor options that you want to use. You can specify multiple options, separated by colons (:). For example: 'max_queue_size=4096:schedule_delay_millis=5000:max_export_batch_size=4096' .

      For more information about the OpenTelemetry batch span processor options, see the OpenTelemetry documentation.

    For example:
    
ResourceManagers:
    OpenTelemetryManager:
       openTelemetryEnabled: true                           # Enable or disable OpenTelemetry tracing. Available on Linux/x86-64. 
       openTelemetryServiceName:                            # Override the Service Name attribute for telemetry spans. 
                                                            # Defaults to integration server name.
       openTelemetryHostName:                               # Override the value of the Hostname attribute for telemetry spans.
       exporterOtlpGrpcEndpoint: 'localhost:4317'           # The GRPC endpoint to which the OpenTelemetry span data is sent. 
                                                            # For example, 'telemetryagent.host.com:4317'.
       exporterOtlpHttpUrl: ''                              # The HTTP/JSON URL to which the OpenTelemetry span data is sent. 
                                                            # For example, 'http://<hostname>:<port>/v1/traces'.
       #openTelemetryHttpSecurityId: ''                     # Specify an opentelemetry credential used in the HTTP exporter.
       #openTelemetryGrpcSecurityId: ''                     # Specify an opentelemetry credential used in the GRPC exporter.
       openTelemetryTruststoreType: ''                      # Type of OpenTelemetry truststore. Valid values are PKCS12, PEM, or JKS.
                                                            # Used only for the GRPC exporter. 
       openTelemetryTruststoreFile: ''                      # Location of the OpenTelemetry truststore. Used only for GRPC exporter.
       openTelemetryTruststorePass: 'truststore::password'  # Credential or resource alias containing the truststore password.
       openTelemetryTrustAlias: ''                          # Alias of the trust certificate in the PKCS12 or JKS truststore.
       openTelemetrySpanProcessor: 'batch'                  # OpenTelemetry span processor. Set to either 'batch' or 'simple'. 
                                                            # Default is 'batch'.
       openTelemetryBatchSpanOptions: 'max_queue_size=4096' # OpenTelemetry batch span processor options, separated by colons (:). 
                                                            # For example, 'max_queue_size=4096:max_export_batch_size=4096'
  3. Save the changes and restart the integration server for the changes to take effect.
  4. You can check the OpenTelemetry settings that are active for an integration server by using one of the following methods:
    • Use the administration REST API to issue a GET on /apiv2/resource-managers/opentelemetry-manager for independent integration servers
    • Run the mqsireportproperties command using the object name opentelemetry-manager on a managed integration server.