IBM App Connect Enterprise Tracing Support (IBM-ACE-TRACING-USER-EXIT)

Content

Download location

Download the IBM ACE Tracing user exit .tgz file from the artifactory. To download the file, use _ as the username and a valid agent key as the password.

Fix download for Linux

The following table lists components, platforms, and file names that apply to this readme file.

Prerequisites and co-requisites

General description

This fix provides IBM App Connect Enterprise tracing capability. You can download user exit implementation and configure your IBM App Connect Enterprise integration server to enable tracing. The tracing data is sent to Instana.

Platforms and prerequisites

This component supports the following platforms:

• AIX 7.2 and 7.3 (POWER Big Endian)
• RHEL 7, 8, and 9 (x86-64, POWER Little Endian, and IBM z Systems)
• SLES 12 and 15 (x86-64, POWER Little Endian, and IBM z Systems)
• Ubuntu 16.04, 18.04, 20.04, and 22.04 LTS (x86-64, POWER Little Endian, and IBM z Systems)
• Windows Server 2012 R2, 2016, 2019, and 2022 (x86-64)

Supported node types

The Instana ACE Tracing user exit supports only the following node types:

• HTTP request
• IBM MQ request
• Kafka request

If you want to enable CICS request support in the IBM Cloud Pak for Integration environment, complete the following steps:

1. Build a customized docker image with ACE Tracing user exit enabled.
2. Deploy ACE application based on this docker image.

IBM MQ message requirements

• The Instana ACE Tracing user exit supports IBM MQ messages that include only an MQRFH2 header because the trace information is written into the IBM MQ message's MQRFH2 header to propagate the trace context. In some IBM MQ consumer clients, the presence of extra header data in IBM MQ messages can cause message processing errors and message rejection.
• Test IBM MQ consumer clients in a nonproduction environment before you enable it in a production environment.
• If you enable trace correlation support and it causes errors in IBM MQ client applications, do one of the following actions:
  • If the IBM MQ consumer client is an application that can be changed, update the application to ignore the additional IBM MQ header data that is added by Instana. If you need assistance to make IBM MQ client changes, contact IBM MQ support.
  • If the IBM MQ consumer client is an application that cannot be changed, do not enable IBM ACE trace correlation support.

Other prerequisites

The Instana ACE Tracing user exit supports only message flows with input node as the entry point.

Before you install and configure the Instana ACE Tracing user exit, install and configure IBM ACE.

Known limitations

• If HTTPS is enabled for REST API interface, you need to specify keystore and keystorePassword parameters in <agent_install_dir>/etc/instana/configuration.yaml. For keystore type, only JKS or P12 is supported.
• The ACE user exit tracing method supports only single-node setups. High Availability (HA) setups and clustered environments are not supported. For HA deployments or clustered environments, use native OpenTelemetry tracing support instead of the user exit method.

Known issues

None.

Setting up ACE tracing

Before you set up ACE tracing, ensure that you use the tracing solution that is supported for your ACE deployment type and version.

• Traditional ACE environment: Select the tracing solution specific to ACE deployment as outlined in the following table:

• IBM Cloud Pak for Integration environment: Use the tracing solution specific to ACE deployment as outlined in the following table:

Enabling tracing for ACE versions earlier than 12.0.7 in the traditional ACE environment

Installing

To download the IBM ACE Tracing user exit, complete the following steps:

1. Download the IBM ACE Tracing user exit .tgz file from artifactory. See Download location. To download the file, use _ as the username and a valid agent key as the password.
2. Extract the downloaded .tgz file to a temporary location.
3. After extraction, find five user-exit packages for different platforms in the directory.
4. Transfer the user exit package specific to a platform on your IBM ACE host.
5. Extract the user exit package into the following directory on your ACE host:
   • Linux and AIX: /var/mqsi/shared-classes
   • Windows: C:\ProgramData\IBM\MQSI\shared-classes
6. Place the following files in your shared classes directory:
   • ACEOpenTracingUserExit.lel: This file contains the Instana ACE user exit, which intercepts the HTTP request, IBM MQ request, and Kafka request, and starts the wrapped OpenTelemetry C++ client library to create spans.
   • tracelibrary.so: This file specifies the wrapped OpenTelemetry C++ client, which provides functions to manage the lifecycle of spans, and sends spans to the target-tracing system.
   • acetracingexit.conf: This configuration file specifies the log level and information about connecting to the host agent.
7. If the ACE server is not installed with global installation, the /var/mqsi/shared-classes or C:\ProgramData\IBM\MQSI\shared-classes directory does not exist on your IBM ACE host. Create the directory /opt/acetracingexit or C:\acetracingexit manually on your IBM ACE host, and extract the .tar file into the directory.

Configuring

To configure your IBM App Connect Enterprise to enable tracing, see Configuring user exit and Configuring IBM App Connect Enterprise tracing.

Configuring user exit

To enable tracing for IBM ACE, complete the following steps:

Note: All the following commands are taken for a Linux or AIX platform. If you run these commands on a Windows platform, replace the directory path /var/mqsi/shared-classes with C:\ProgramData\IBM\MQSI\shared-classes.

1. Stop the integration node.

   mqsistop <integrationnodename>

2. Install the user exit on an integration node by setting the UserExitPath property that uses the mqsichangeflowuserexits command.

   mqsichangeflowuserexits <integrationnodename> -o -x /var/mqsi/shared-classes

   If you extract IBM ACE tracing .tar files into /opt/acetracingexit directory, replace /var/mqsi/shared-classes with /opt/acetracingexit.

3. Activate the user exit.

   User exits can be active or inactive, and are inactive by default. You can activate user exit for an integration node, an integration server, or a specific message flow.

   1. Activate the user exit for an integration node.

      1. Activate the user exit:

         mqsichangeflowuserexits <integrationnodename> -o -a ACEOpenTracingUserExit

      2. Verify the user exit:

         mqsireportflowuserexits <integrationnodename> -o

         See the following sample output:

         mqsireportflowuserexits BK3 -o
         BIP8854I: User Exits active for integration server 'BK3': ACEOpenTracingUserExit.
         BIP8855I: User Exits inactive for integration server 'BK3': .
         BIP8741I: User Exit path for integration server 'BK3': /var/mqsi/shared-classes.
         BIP8071I: Successful command completion.

      3. Start the integration node:

         mqsistart <integrationnodename>

   2. Activate the user exit for an integration server.

      1. Start the integration node:

         mqsistart <integrationnodename>

      2. Activate the user exit:

         mqsichangeflowuserexits <integrationnodename> -e <integrationservername> -a ACEOpenTracingUserExit

   3. Activate the user exit for a message flow.

      1. Start the integration node:

         mqsistart <integrationnodename>

      2. Activate the user exit for a message flow:

         mqsichangeflowuserexits <integrationnodename> -e <integrationservername> -k <applicationname> -f <messageflow> -a ACEOpenTracingUserExit

   4. Repeat the steps for other integration nodes, integration servers, or message flows for which you want to activate the user exit.

For more information, see the following links:

• IBM App Connect Enterprise - Deploying a user exit
• IBM App Connect Enterprise - mqsichangeflowuserexits command - Windows, Linux, and AIX systems

Configuring IBM App Connect Enterprise tracing

1. Go to /var/mqsi/shared-classes directory.

2. Edit acetracingexit.conf file.

   # configuration for ace tracing exit
   LOG_LEVEL="info" #Log level: info, warn, error, debug
   SPAN_FORMAT="instana"
   CICS_SUPPORT="off" #Propagate trace context for CICS request: off, on
   MONITOR_LEVEL="verbose" #ACE tracing level: off, normal, verbose
   INSTANA_AGENT_HOST="localhost" #(optional)
   INSTANA_AGENT_PROTO="http" #(optional)
   HOST_ALIAS="<YOUR-HOST-NAME>" #(optional)

   where:

   • LOG_LEVEL - Specifies the log level, which can be one of the following types: info, warn, error, or debug. The log file is at the /tmp/trace directory.
   • SPAN_FORMAT - Specifies where the span data is sent to. Set this variable to instana. Instana ACE Tracing user exit sends span data to the host agent endpoint http://localhost:42699 by default. Update the configuration fields INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO if you want to send the span data to a remote host agent that uses HTTPS protocol. You need to have the same SPAN_FORMAT setting for all IBM ACE instances.
   • CICS_SUPPORT - This parameter controls whether the support for tracing CICS request is enabled. Set it to on to enable and off to disable the support.
   • MONITOR_LEVEL - Specifies the tracing level of IBM ACE, which can be one of the following types: off, normal, or verbose. If MONITOR_LEVEL is set to off, no tracing context is appended to the outgoing request. If MONITOR_LEVEL is set to normal, the tracing context is appended only when the IBM MQ message includes the RFH2 header. If MONITOR_LEVEL is set to verbose, the tracing context is appended to all outgoing HTTP or IBM MQ requests.
   • INSTANA_AGENT_HOST - Specifies the agent host where the Instana format span data is sent to. By default, localhost is used. If you specify a remote agent host, you must also add a line http.listen=* in *instanaAgentDir*/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
   • INSTANA_AGENT_PROTO - Specifies the connection type between IBM ACE Tracing user exit and the host agent. By default, http is used. However, https is also supported. If you want to change it to https, you need to follow Set up TLS Encryption for Agent Endpoint to secure the Instana agent endpoint first.
   • HOST_ALIAS - Specifies a host alias for the span data that is collected by Instana ACE Tracing user exit. So, calls to IBM ACE can be linked to the infrastructure entity if the integration node or integration server is also monitored by the IBM ACE sensor. The FQDN of the IBM ACE host is used by default. The host alias value needs to match with the IBM ACE sensor host that is specified in the host agent configuration yaml file. You must specify a host alias only if the FQDN of the IBM ACE host is not used in the Instana ACE sensor configuration and when the host agent is not on the local IBM ACE host. The IBM ACE sensor can discover the FQDN for the local integration nodes or integration servers.
3. Save the file and restart integration node or integration server.

Repeat these installation and configuration steps on other IBM ACE hosts that you want to enable tracing for.

You can view Instana ACE Tracing data in the Instana UI.

Unconfiguring

1. Unconfigure user exit.
   1. Unconfigure user exit for an integration node:

      1. Stop the integration node:

         mqsistop <integrationNodeName>

      2. Deactivate the user exit:

         mqsichangeflowuserexits <integrationNodeName> -o -a ""

      3. Restart the integration nodes.

   2. Deactivate user exit for an integration server:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ""

   3. Deactivate user exit for a message flow:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ""

2. Repeat steps on other integration nodes.

Troubleshooting

The log files are at /var/mqsi/trace on Linux platform. The file name is aceExit*.log.

You can set log level in acetracingexit.conf as described in Configuring IBM App Connect Enterprise tracing section.

Enabling tracing for ACE 12.0.7 or later in the traditional ACE environment

For ACE 12.0.7.0 and later, OpenTelemetry-based tracing is built in, and Instana can ingest the generated tracing data directly. You no longer need to install the Instana ACE Tracing user exit. Do not enable Instana ACE Tracing user exit and ACE OpenTelemetry tracing simultaneously because the tracing calls mix with the upstream or downstream tracing calls. For more information about OpenTelemetry tracing integration, see Configuring OpenTelemetry tracing for integration runtimes.

Integration servers support OpenTelemetry trace on the following platforms:

• AIX (IBM ACE 12.0.10.0 and later)
• Linux x86-64 (IBM ACE 12.0.7.0 and later)
• Linux on System z (IBM ACE 12.0.8.0 and later)
• Linux on Power Systems - Little Endian (IBM ACE 12.0.10.0 and later)
• Windows (IBM ACE 12.0.8.0 and later)

Configuring ACE OpenTelemetry tracing

To enable OpenTelemetry tracing for IBM ACE, follow the steps:

1. Configure OpenTelemetry data ingestion for Instana. For more information, see Configuring OpenTelemetry data ingestion.
2. Enable ACE OpenTelemetry tracing. For more information, see Configuring OpenTelemetry trace for an integration server.
3. Configure correct hostname to enable correct infrastructure correlation between entities.

For more information about considerations and limitations, see OpenTelemetry considerations and limitations.

Enabling tracing in the IBM Cloud Pak for Integration environment

Based on your ACE version, tracing for ACE integration servers in IBM Cloud Pak for Integration (amd64 only) is handled as follows:

• For ACE versions before 12.0.8, the Instana AutoTrace webhook deploys the Instana User Exit for ACE Tracing into the integration server automatically. (Deprecated)
• For ACE 12.0.8 or later, you must enable OpenTelemetry-based tracing solution provided by ACE directly without relying on the Instana AutoTrace webhook. For more information, see Configuring ACE OpenTelemetry tracing. The AutoTrace webhook does not currently support or enable native OpenTelemetry tracing in ACE.

After the ACE Tracing is enabled, click Applications > Services in the Instana UI to check the tracing call details. The service name for the ACE Tracing that is enabled through these options are as follows:

• Instana ACE user exit: The service name includes Pod IP and integration server name of ACE, and the format is <ACE_Pod_IP>-<IntegrationServer_Name>:<IntegrationServer_Name>, for example, 10.254.17.125-is-01-1206:is-01-1206.
• ACE OpenTelemetry: The service name includes the integration server name, and the format is IBM App Connect Enterprise-<IntegrationServer_Name>, for example, IBM App Connect Enterprise-is-01-customer.

Additional information

The Secure Hash Algorithm 1 (SHA256) checksum of the tar file is dd7e52c5c644ea66b266d5676484792338b3840fd1344ea58fb2803a6018cf66

Document change history