Configuring IBM App Connect Enterprise (ACE) Tracer

Supported node types

The Instana ACE Tracing user exit supports only the following node types:

• HTTP request
• IBM MQ request
• CICS request
• Kafka request

In the traditional ACE environment, CICS request is supported only on IBM ACE 12.0.8 and later.

If you want to enable CICS request support in the IBM Cloud Pak for Integration environment, complete the following steps:

1. Build a customized docker image with the ACE Tracing user exit enabled.
2. Deploy the ACE application based on this docker image.

IBM MQ message requirements

• The Instana ACE Tracing user exit supports IBM MQ messages that include only an MQRFH2 header because the trace information is written into the IBM MQ message's MQRFH2 header to propagate the trace context. In some IBM MQ consumer clients, the presence of extra header data in IBM MQ messages can cause message processing errors and message rejection.

• Test IBM MQ consumer clients in a nonproduction environment before you enable it in a production environment.

• If you enable trace correlation support and it causes errors in IBM MQ client applications, do one of the following actions:

  • If the IBM MQ consumer client is an application that can be changed, update the application to ignore the additional IBM MQ header data that is added by Instana. If you need assistance to make IBM MQ client changes, contact IBM MQ support.
  • If the IBM MQ consumer client is an application that cannot be changed, do not enable IBM ACE trace correlation support.

Others

The Instana ACE Tracing user exit supports only message flows with input node as the entry point.

Before you install and configure the Instana ACE Tracing user exit, install and configure IBM ACE.

Downloading the IBM ACE Tracing user exit

To download the IBM ACE Tracing user exit, complete the following steps:

1. Download the IBM ACE Tracing user exit .tgz file from artifactory. To download the file, use _ as the username and a valid agent key as the password.

2. Extract the downloaded .tgz file to a temporary location.

3. After extraction, find five user-exit packages for different platforms in the directory.

4. Transfer the user exit package specific to a platform on your IBM ACE host.

5. Extract the user exit package into the following directory on your ACE host.

   • Linux and AIX: /var/mqsi/shared-classes
   • Windows: C:\ProgramData\IBM\MQSI\shared-classes

   Place the following files in your shared classes directory:

   • ACEOpenTracingUserExit.lel: This file contains the Instana ACE user exit, which intercepts the HTTP request, IBM MQ request, and Kafka request, and starts the wrapped OpenTelemetry C++ client library to create spans.
   • tracelibrary.so: This file specifies the wrapped OpenTelemetry C++ client, which provides functions to manage the lifecycle of spans, and sends spans to the target-tracing system.
   • acetracingexit.conf: This configuration file specifies the log level and information about connecting to the host agent.

   If the ACE server is not installed with global installation, the /var/mqsi/shared-classes or C:\ProgramData\IBM\MQSI\shared-classes directory does not exist on your IBM ACE host. Create the directory /opt/acetracingexit or C:\acetracingexit manually on your IBM ACE host, and extract the .tar file into the directory.

Configuring user exit

To enable tracing for IBM ACE, complete the following steps:

All the following commands are taken for a Linux or AIX platform. If you run these commands on a Windows platform, replace the directory path /var/mqsi/shared-classes with C:\ProgramData\IBM\MQSI\shared-classes.

1. Stop the integration node.

   mqsistop <integrationNodeName>
   

2. Install the user exit on an integration node by setting the UserExitPath property that uses the mqsichangeflowuserexits command.

   mqsichangeflowuserexits <integrationNodeName> -o -x /var/mqsi/shared-classes
   

   If you extract IBM ACE tracing .tar files into /opt/acetracingexit directory, replace /var/mqsi/shared-classes with /opt/acetracingexit.

3. Activate the user exit.

   User exits can be active or inactive, and are inactive by default. You can activate user exit for an integration node, an integration server, or a specific message flow.

   1. Activate the user exit for an integration node.

      1. Activate the user exit:

         mqsichangeflowuserexits <integrationNodeName> -o -a ACEOpenTracingUserExit
         

      2. Verify the user exit:

         mqsireportflowuserexits <integrationNodeName> -o
         

         See the following sample output:

         # mqsireportflowuserexits BK3 -o
         BIP8854I: User Exits active for integration server 'BK3': ACEOpenTracingUserExit.
         BIP8855I: User Exits inactive for integration server 'BK3': .
         BIP8741I: User Exit path for integration server 'BK3': /var/mqsi/shared-classes.
         BIP8071I: Successful command completion.
         

      3. Start the integration node:

         mqsistart <integrationNodeName>
         

   2. Activate the user exit for an integration server.

      1. Start the integration node:

         mqsistart <integrationNodeName>
         

      2. Activate the user exit:

         mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ACEOpenTracingUserExit
         

   3. Activate the user exit for a message flow.

      1. Start the integration node:

         mqsistart <integrationNodeName>
         

      2. Activate the user exit for a message flow:

         mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ACEOpenTracingUserExit
         

   4. Repeat the steps for other integration nodes, integration servers, or message flows for which you want to activate the user exit.

For more information, see the following links:

• IBM ACE: Deploying a user exit
• IBM ACE: mqsichangeflowuserexits command - Windows, Linux, and AIX systems

Configuring Instana ACE Tracing

1. Go to the /var/mqsi/shared-classes directory.

2. Edit the acetracingexit.conf file:

   # configuration for ace tracing exit
   LOG_LEVEL="info" #Log level: info, warn, error, debug
   SPAN_FORMAT="instana"
   CICS_SUPPORT="off" #Propagate trace context for CICS request: off, on
   MONITOR_LEVEL="verbose" #ACE tracing level: off, normal, verbose
   INSTANA_AGENT_HOST="localhost" #(optional)
   INSTANA_AGENT_PROTO="http" #(optional)
   HOST_ALIAS="<YOUR-HOST-NAME>" #(optional)
   
   

   where:

   • LOG_LEVEL
     Specifies the log level, which can be one of the following types: info, warn, error, or debug. The log file is at the /tmp/trace directory.
   • SPAN_FORMAT
     Specifies where the span data is sent to. Set this variable to instana. Instana ACE Tracing user exit sends span data to the host agent endpoint http://localhost:42699 by default. Update the configuration fields INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO if you want to send the span data to a remote host agent that uses HTTPS protocol. You need to have the same SPAN_FORMAT setting for all IBM ACE instances.
   • CICS_SUPPORT
     This parameter controls whether the support for tracing CICS request is enabled. Set it to on to enable and off to disable the support.
   • MONITOR_LEVEL
     Specifies the tracing level of IBM ACE, which can be one of the following types: off, normal, or verbose. If MONITOR_LEVEL is set to off, no tracing context is appended to the outgoing request. If MONITOR_LEVEL is set to normal, the tracing context is appended only when the IBM MQ message includes the RFH2 header. If MONITOR_LEVEL is set to verbose, the tracing context is appended to all outgoing HTTP or IBM MQ requests.
   • INSTANA_AGENT_HOST
     Specifies the agent host where the Instana format span data is sent to. By default, localhost is used. If you specify a remote agent host, you must also add a line http.listen=* in <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
   • INSTANA_AGENT_PROTO
     Specifies the connection type between IBM ACE Tracing user exit and the host agent. By default, http is used. However, https is also supported. If you want to change it to https, you need to follow Set up TLS Encryption for Agent Endpoint to secure the Instana agent endpoint first.
   • HOST_ALIAS
     Specifies a host alias for the span data that is collected by Instana ACE Tracing user exit. So, calls to IBM ACE can be linked to the infrastructure entity if the integration node or integration server is also monitored by the IBM ACE sensor. The FQDN of the IBM ACE host is used by default. The host alias value needs to match with the IBM ACE sensor host that is specified in the host agent configuration yaml file. You must specify a host alias only if the FQDN of the IBM ACE host is not used in the Instana ACE sensor configuration and when the host agent is not on the local IBM ACE host. The IBM ACE sensor can discover the FQDN for the local integration nodes or integration servers.

3. Save the file and restart integration node or integration server.

Repeat these installation and configuration steps on other IBM ACE hosts that you want to enable tracing for.

You can view Instana ACE Tracing data in the Instana UI.

Unconfiguring Instana ACE Tracing user exit

1. Unconfigure user exit.

   1. Unconfigure user exit for an integration node:

      1. Stop the integration node:

         mqsistop <integrationNodeName>
         

      2. Deactivate the user exit:

         mqsichangeflowuserexits <integrationNodeName> -o -a ""
         

      3. Restart the integration nodes.

   2. Deactivate user exit for an integration server:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ""
      

   3. Deactivate user exit for a message flow:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ""
      

2. Repeat steps on other integration nodes.

Configuring ACE native tracing

To enable native tracing for IBM ACE, follow the steps:

1. Configure OpenTelemetry data ingestion for Instana. For more information, see Configuring OpenTelemetry data ingestion.

2. Enable ACE native tracing. For more information, see Configuring OpenTelemetry trace for an integration server.

For more information about considerations and limitations, see OpenTelemetry considerations and limitations.