Monitoring end-to-end transactions in IBM webMethods Hybrid Integration

You can configure IBM® App Connect Enterprise to send data for end-to-end monitoring in IBM webMethods Hybrid Integration. Configure the server.conf.yaml file for your integration server and provide end-to-end monitoring credentials in a vault.

To configure an App Connect Enterprise integration server for end-to-end monitoring, complete the following steps.

1. Copy the API key and OpenTelemetry endpoint from the End-to-end monitoring page in IBM webMethods Hybrid Integration.
   You use the API key to authenticate with the end-to-end monitoring OpenTelemetry receiver endpoint.
   1. In the webMethods Hybrid Integration main menu (), click End-to-end monitoring.
   2. In the End-to-end monitoring navigation pane, click Manage, then click API key.
   3. If the API key exists, click Show API key, then click Copy . If no API key exists, click Generate. Retain the API key for a later step.
   4. Copy the OTel EndPoint URL and retain it for a later step.
2. In App Connect Enterprise, create a work directory for your integration server by running the mqsicreateworkdir command.
   For more information, see mqsicreateworkdir command.
3. In App Connect Enterprise, create a vault by running the mqsivault command.
   For example, the following command creates a vault in the current work directory. The vault key must be at least 8 characters long.

   mqsivault –-work-dir . –-vault-key <vaultKey> --create 

   For more information, see mqsivault command.
4. In App Connect Enterprise, encrypt and store the end-to-end monitoring credentials in the vault by using the mqsicredentials command.
   For example, the following command stores the username and password for end-to-end monitoring in the vault that you created in the previous step.

   mqsicredentials --work-dir . --vault-key <vaultKey> --create --credential-type opentelemetry --credential-name <credName> --username <envID> --password <APIkeyfromE2E> 

   • You use the credential-name property to identify this set of credentials when you update the server.conf.yaml file for the integration server.
   • The username is the alphanumeric environment ID from your webMethods Hybrid Integration instance. To find your environment ID, go to the webMethods Hybrid Integration main menu and click Environment management. You can also find this environment ID after https:// in the URL of your webMethods Hybrid Integration instance. For example, if the URL of your webMethods Hybrid Integration instance is https://dev1234567.a-vir-b1.platform.ipaas.automation.ibm.com/, the environment ID is dev1234567.
   • The password is the API key that you copied from end-to-end monitoring in step 1.

   For more information, see mqsicredentials command.
5. In App Connect Enterprise, update the EnvironmentVariables and OpenTelemetryManager sections of the server.conf.yaml file with your webMethods Hybrid Integration environment ID and end-to-end monitoring credentials.
   • In the EnvironmentVariables section, set MQSI_TENANT_AUTH_URL and MQSI_TENANT_ID to the environment ID for your webMethods Hybrid Integration instance.
   • In the OpenTelemetryManager section, set openTelemetryEnabled to true.
   • Set exporterOtlpGrpcEndpoint to the OTel EndPoint URL that you copied in step 1.
   • Set openTelemetryGrpcSecurityId to the value of the credential name that you specified in the mqsicredentials command.

   For more information, see Configuring an integration server by modifying the server.conf.yaml file.

   EnvironmentVariables:
      MQSI_TENANT_AUTH_URL: 'envID'
      MQSI_TENANT_ID: 'envID'
   OpenTelemetryManager:
      openTelemetryEnabled: true
      exporterOtlpGrpcEndpoint: 'https://otel.a-vir-r1.e2em.ipaas.automation.ibm.com:4317' 
      openTelemetryGrpcSecurityId: 'credName'

6. Start the configured integration server.
   For example, run the following IntegrationServer command with the vault key for your integration server vault.

   IntegrationServer --work-dir . --vault-key <vaultKey> 

   For more information, see IntegrationServer command.

   Troubleshooting:

   /usr/share/grpc/roots.pem failed to load or wasn't found

   In some circumstances, you might see an error message in the integration server console during startup that states that the /usr/share/grpc/roots.pem file failed to load or wasn't found. Further error messages are issued when transactions that you want to monitor are processed, such as No root certs in config and Failed to create secure subchannel.

   To resolve these errors, use the command console to set the environment variable GRPC_DEFAULT_SSL_ROOTS_FILE_PATH to an empty path. For example, set GRPC_DEFAULT_SSL_ROOTS_FILE_PATH = "". By setting this environment variable to an empty string, the integration server doesn't need the GRPC root certificate when it starts.

   SSL certificate issues for OpenTelemetry Protocol (OTLP) export

   If you configure App Connect Enterprise to export OpenTelemetry traces by using OTLP over gRPC, you might encounter SSL handshake errors because of missing trusted certificates.

   The following steps describe how to resolve these issues.

   1. Use the following command to retrieve the certificate chain:

      openssl s_client -connect <otlp-endpoint>:port -showcerts

   2. Identify and save the root certificate authority (CA) or intermediate CA certificate from the output.
   3. Configure App Connect Enterprise to trust the certificate by either importing the certificate into the truststore or setting an environment variable. To use an environment variable, save the certificate to a file, then set the environment variable in the EnvironmentVariables section of the server.conf.yaml file.

      EnvironmentVariables:
          OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE: /path/to/issuer.crt

   4. Restart the integration server.