Monitoring IBM App Connect Enterprise (ACE)

Instana provides a comprehensive monitoring solution for your IBM ACE environment, giving you end-to-end visibility into its performance. After you install the Instana host agent, you must configure the IBM ACE sensor. Then, you can view metrics and end-to-end traces of all requests in your IBM ACE environment in the Instana UI.

Supported information

Supported operating systems

The IBM ACE sensor is automatically installed after the host agent is installed on supported operating systems. For more information, see the supported operating systems for host agents.

The automatic discovery feature of the IBM ACE sensor in the traditional ACE environment supports the following OS:

  • Linux
  • AIX

Windows is not supported.

Supported ACE versions and platforms

  • The IBM ACE sensor supports metrics and configuration data for IBM Integration Bus 10.0.x, IBM ACE 11.0.x, and IBM ACE 12.0.x that are installed in all supported operating systems. The automatic discovery feature of the IBM ACE sensor in the traditional ACE environment supports IBM ACE 11 or later. IBM Integration Bus (IIB) 10 is not supported.

  • IBM ACE Tracing supports different versions and platforms. For more information, see Supported IBM ACE Tracing versions and platforms.

    Notes:
    • For IBM Integration Bus 10, make sure that you use version 10.0.0.24 or later. Or else, the Number of Thread In Pool metric does not report the correct value in accordance with the WLM policy.
    • In a cloud-native environment, only IBM ACE certified container image is supported.

Other supported information

The IBM ACE sensor supports both local and remote monitoring.

Configuring the IBM ACE sensor

You can configure the IBM ACE sensor in a traditional or cloud-native environment.

Traditional ACE environment

Prerequisites

Before you configure the IBM ACE sensor, complete the following steps:

  • If the server is IIB 10, you need to run some commands to check the statistics state. You can turn on or turn off statistics collection when needed.

    The metrics return only when the statistics gathering and resource statistics collection are active.

    • To check the collection state of resource statistics for specified server in specified integration node, run the following command:

      mqsireportresourcestats integrationNode -e integrationServer

    • If the collection state is inactive, turn on the resource statistics collection by running the following command:

      mqsichangeresourcestats integratorNode -e integrationServer -c active

      For more information, see mqsireportresourcestats and mqsichangeresourcestats.

    Notes:
    • If you want to check the thread use of each message flow, you need to enable thread data with option -t basic when you are using the mqsichangeflowstats command.
    • You can request two types of data collection, snapshot and archive. For more information, see Message flow accounting and statistics collection options. It is good to use snapshot type for IBM ACE Sensor.

  • If the monitored server is an independent integration server, confirm the following items:

    • Configure the publication of event messages. The publication is disabled by default and the ACE sensor monitors only OperationalEvents, not BusinessEvents. For more information, see Configuring the publication of event messages.
    • Use ACE 11.0.0.11 or later. Otherwise, you cannot use the UNIX domain socket, since .uds file does not exist for each server.

  • If IBM MQ is used for ACE/IIB traditional environment, set up a channel. If the integration node or server is associated with a queue manager, you must set up a server connection channel. The performance metrics data for integration server, message flow, and flow node flows through this channel. This channel is accessible by the username or password that is specified in the configuration.yaml file. In addition to the channel name, username, and password, you also need to specify the queue manager name and listening port in configuration.yaml. If you have any issue about channel configuration, refer to Resolving CHLAUTH access issues.

    • If the state is inactive, turn on the snapshot statistics collection for all message flows in the specified integration server for specified integration node and emit the data in JSON format.

      • To turn on the snapshot statistics collection without setting thread data level, run the following command:

        mqsichangeflowstats integrationNode -s -e integrationServer -j -c active -o json

      • To turn on the snapshot statistics collection with setting thread data level, run the following command:

        mqsichangeflowstats integrationNode -s -e integrationServer -j -t basic -c active -o json

      • To turn on the snapshot statistics collection with setting both thread data level and node data level:

        mqsichangeflowstats integrationNode -s -e integrationServer -j -t basic -n basic -c active -o json

        For more information, see mqsireportflowstats and mqsichangeflowstats.

      Notes:
      • If you want to check the thread use of each message flow, enable the thread data with option -t basic when you use the mqsichangeflowstats> command.
      • You can request two types of data collection, snapshot and archive. For more information, see Message flow accounting and statistics collection options. The snapshot type is the preferred choice for IBM ACE Sensor.

  • If IBM MQ is used for ACE or IIB traditional environment, set up a channel. If the integration node or server is associated with a queue manager, you must set up a server connection channel. The performance metrics data for integration server, message flow, and flow node flows through this channel. This channel is accessible by the username or password that is specified in the configuration.yaml file. In addition to the channel name, username, and password, you also need to specify the queue manager name and listening port in configuration.yaml. For more information about channel configuration, see Resolving CHLAUTH access issues.

  • Ensure that the following requirements are met to enable the automatic discovery feature in the traditional ACE environment:

    • The IBM ACE sensor is enabled without configuring integration node or server in the <agent_install_dir>/etc/instana/configuration.yaml file.
    • Permission to access the WebSphere Message Broker work path (for example, /var/mqsi/) is available to start the agent.
    • The MQTT process is running properly for the related integration node. To check, run the **ps -ef | grep -E 'bipMQTT.*<INTEGRATION_NODE>’ **command on the host.

    With automatic discovery, you can see all running integration servers that are listed on the host dashboard of the Instana UI even when all the prerequisites are not met.

Procedure

For the traditional ACE environment, Instana supports monitoring of both remote and local IBM ACE instances. You need to configure the fields in the agent configuration <agent_install_dir>/etc/instana/configuration.yaml file. You can use the following configurations:

  • To use remote monitoring, set the IP address of the target ACE server as the value of the host. Or else, comment out the host attribute and the local monitoring is used by default.

  • To monitor brokers or integration servers with the same name on different hosts, use brokerName:x or integrationServer:x as the key of the brokers or integration servers, such as BK1:1and BK:2.

  • To use automatic discovery feature, do not set any field under NodesOrServers. The automatic discovery feature in the ACE sensor discovers all integration servers that are running on the host and starts monitoring them automatically without any configuration.

The minimum poll rate depends on IBM ACE. If the poll rate that you set on Instana is lesser than the minimum poll rate on IBM ACE, the minimum poll rate is applied.

To get tips about how to get the values for these fields, see the Useful tips for ACE sensor configuration blog.

See the following sample configuration.yaml file:

com.instana.plugin.ace:
  enabled: true
  poll_rate: 60                  # Default value is 60 seconds, and the minimum value is 20 seconds.
  forceRemote: false             # Force local monitoring to remote monitoring
  flowNodesExcludedRegex: ''     # Global regex for filtering exclusive flow nodes from all integration nodes (optional)
  NodesOrServers:                # Multiple Integration node instances or multiple standalone Integration servers can be specified
    BK1:  # Example1: Integration Node with MQTT (required)
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      ######################################################################################################
      # host: 'aceserver.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'            # ACE server's rest api port (required)
      #aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      #acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      #keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      #keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      #excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      #flowNodesExcludedRegex: ''    # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      #availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      # Note: if MQTT is used, then keep mqport only and comment out other MQ parameters.
      #######################################################################################################
      #mqHost: 'mqserver.com'        # MQ host/IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '11883'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      #queuemanagerName: 'QM1'       # Queue Manager name (required for IBM MQ)
      #channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      #mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      #mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
    IS1:  # Example2: High Availability integration Node with MQ configured. (required)
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      ######################################################################################################
      host: 'ha-instance1.com,ha-instance2.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'            # ACE server's rest api port (required)
      #aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      #acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      #keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      #keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      #excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      #flowNodesExcludedRegex: ''    # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      #availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      #######################################################################################################
      #mqHost: 'mqserver.com'       # MQ host or IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '1414'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM2'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
    BK1:2:  # Example3: Same broker names on different hosts, such as BK1, BK1:1, BK1:2, BK1:n, etc.
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the \u2018#\u2019 when needed.
      ######################################################################################################
      host: 'aceserver.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'           # ACE server's rest api port (required)
      aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      # keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      # keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      # excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      # flowNodesExcludedRegex: ''  # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      # availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the \u2018#\u2019 when needed.
      # Note: if MQTT is used, then keep mqport only and comment out other MQ parameters.
      #######################################################################################################
      mqHost: 'mqserver.com'        # MQ host/IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '1414'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM1'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)

Cloud-native ACE environment

Configuration for the IBM ACE sensor in the cloud-native environment is auto-discovered.

Viewing metrics

To view the metrics, complete the following steps:

  1. From the navigation menu in the Instana UI, select Infrastructure.
  2. Click a specific monitored IBM ACE instance.

You can see the IBM ACE dashboard with all the collected metrics and monitored processes.

Configuration data

  • Default Queue Manager Name
  • Product Name
  • Version
  • Build Level
  • Platform Name
  • Platform Architecture
  • Platform Version
  • Application Name

Performance metrics

Integration server

Metric Description
Initial Heap Memory The initial amount of memory that the JVM requests from the operating system for memory management during startup.
Used Heap Memory The amount of memory that is in use.
Committed Heap Memory The amount of memory that is allocated to the JVM by the operating system.
Max Heap Memory The maximum amount of memory that can be used for memory management.
Initial Nonheap Memory The initial amount of memory that the JVM requests from the operating system for memory management during startup.
Used Nonheap Memory The amount of memory that is in use.
Committed Nonheap Memory The amount of memory that is allocated to the JVM by the operating system.
Max Nonheap Memory The maximum amount of memory that can be used for memory management.
Cumulative GC Time The time when the data is collected, in seconds.
Cumulative Number of GC The total number of garbage collections occurred for this instance of the JVM.

Message flow

Metric Data type Description
Total Elapsed Time Numeric The total elapsed time in microseconds that a message flow spent in processing input messages.
Maximum Elapsed Time Numeric The maximum elapsed time in microseconds that a message flow spent in processing an input message.
Minimum Elapsed Time Numeric The minimum elapsed time in microseconds that a message flow spent in processing an input message.
Total CPU Time Numeric The total CPU time in microseconds that a message flow spent in processing an input message.
Max CPU Time Numeric The maximum CPU time in microseconds that a message flow spent in processing an input message.
Min CPU Time Numeric The minimum CPU time in microseconds that a message flow spent in processing an input message.
CPU Time Waiting For Input messages Numeric The total CPU time in microseconds that a message flow spent in waiting for input messages.
Elapsed Time Waiting For Input messages Numeric The total elapsed time in microseconds that is spent in waiting for input messages.
Num of All Input messages Numeric The total number of input messages.
Total Size of Input Messages Numeric The total size of input messages in bytes.
Maximum Size of Input Messages Numeric The maximum size of input messages in bytes.
Minimum Size of Input Messages Numeric The minimum size of input messages in bytes.
Number of Threads in Pool Numeric The number of threads in the pool.
Number of Times That Max Threads Reached Numeric The number of times the maximum number of threads is reached.
Number of MQGET Errors Numeric The number of MQGET errors for MQInput nodes or Web Services errors for HTTPInput nodes.
Number of Messages with Errors Numeric The number of messages that contain errors.
Number of Errors when you are processing Messages Numeric The number of errors that occur when you process a message.
Number of Transaction timeouts Numeric The number of transaction timeouts that occur when you process a message (for AggregateReply nodes only).
Number of Transaction Commits Numeric The number of transaction commits that occur when you process a message.
Number of Transaction Backouts Numeric The number of transaction backouts that occur when you process a message.
Thread Utilization Numeric The number of thread usage for each message flow.

Message flow node

Metric Data type Description
Total Elapsed Time Numeric The total elapsed time in microseconds that is spent in processing input messages.
Max Elapsed Time Numeric The maximum elapsed time in microseconds that is spent in processing input messages.
Min Elapsed Time Numeric The minimum elapsed time in microseconds that is spent in processing input messages.
Total CPU Time Numeric The total CPU time in microseconds that is spent in processing input messages.
Max CPU Time Numeric The maximum CPU time in microseconds that is spent in processing an input message.
Min CPU Time Numeric The minimum CPU time in microseconds that is spent in processing an input message.
Number of Invocations Numeric The total number of messages that are processed by this flow node.
Number of Input-type Terminals Numeric The number of input-type terminals.
Number of Output-type Terminals Numeric The number of output-type terminals.

Known limitations

If HTTPS is enabled for REST API interface, you need to specify keystore and keystorePassword parameters in <agent_install_dir>/etc/instana/configuration.yaml. For keystore type, only JKS or P12 is supported.

Health signatures

Each sensor has a curated knowledgebase of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents, which depend on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any entity.

For more information about built-in events for IBM ACE sensor, see Built-in events reference.

Troubleshooting

You might encounter some monitoring issues with Instana. For more information about how to resolve them, see Troubleshooting.

Tracing

Supported platforms

The Instana ACE Tracing user exit supports the following platforms:

  • IBM ACE 11.0.0.8 or later.
  • Traditional ACE in Linux x86_64, Linux ppc64le, Linux s390x, AIX 7.2, AIX 7.3, Windows, and container ACE in IBM Cloud Pak for Integration (amd64 only).
Notes:

Prerequisites

Supported node types

The Instana ACE Tracing user exit supports only HTTP request, IBM MQ request, CICS request, and Kafka request. In the traditional ACE environment, CICS request is supported only on IBM ACE 12.0.8 and later. If you want to enable CICS request support in the IBM Cloud Pak for Integration environment, you must build a customized docker image with ACE Tracing user exit enabled. Then, deploy the ACE application based on this docker image.

IBM MQ message requirements

IBM MQ messages that include only an MQRFH2 header are supported because the trace information is written into the IBM MQ message's MQRFH2 header to propagate the trace context. In some IBM MQ consumer clients, the presence of extra header data in IBM MQ messages can cause message processing errors and message rejection.

Important: You must test IBM MQ consumer clients in a *nonproduction environment* before you enable it in a production environment. If on enabling trace correlation support causes errors in IBM MQ client applications, do one of the following actions:
  • If the IBM MQ consumer client is an application that can be changed, update the application to ignore the additional IBM MQ header data that is added by Instana. You can contact IBM MQ support if assistance is needed to make IBM MQ client changes.
  • If the IBM MQ consumer client is an application that cannot be changed, *do not enable* IBM ACE trace correlation support.

Others

The Instana ACE Tracing user exit supports only message flows with input node as the entry point.

IBM ACE must be installed and configured before you install and configure this component.

Enabling tracing for ACE versions earlier than 12.0.7 in the traditional ACE environment

Downloading the IBM ACE Tracing user exit

To download the IBM ACE Tracing user exit, complete the following steps:

  1. Download the IBM ACE Tracing user exit .tgz file from artifactory. To download the file, use _ as the username and a valid agent key as the password.

  2. Extract the downloaded .tgz file to a temporary location.

  3. After extraction, find five user-exit packages for different platforms in the directory.

  4. Transfer the user exit package specific to a platform on your IBM ACE host.

  5. Extract the user exit package into the following directory on your ACE host.

    • Linux and AIX: /var/mqsi/shared-classes
    • Windows: C:\ProgramData\IBM\MQSI\shared-classes

    Place the following files in your shared classes directory:

    • ACEOpenTracingUserExit.lel: This file contains the Instana ACE user exit, which intercepts the HTTP request, IBM MQ request, and Kafka request, and starts the wrapped OpenTelemetry C++ client library to create spans.
    • tracelibrary.so: This file specifies the wrapped OpenTelemetry C++ client, which provides functions to manage the lifecycle of spans, and sends spans to the target-tracing system.
    • acetracingexit.conf: This configuration file specifies the log level and information about connecting to the host agent.

    If the ACE server is not installed with global installation, the /var/mqsi/shared-classes or C:\ProgramData\IBM\MQSI\shared-classes directory does not exist on your IBM ACE host. Create the directory /opt/acetracingexit or C:\acetracingexit manually on your IBM ACE host, and extract the .tar file into the directory.

Configuring user exit

To enable tracing for IBM ACE, complete the following steps:

All the following commands are taken for a Linux or AIX platform. If you run these commands on a Windows platform, replace the directory path /var/mqsi/shared-classes with C:\ProgramData\IBM\MQSI\shared-classes.

  1. Stop the integration node.

    mqsistop <integrationNodeName>

  2. Install the user exit on an integration node by setting the UserExitPath property that uses the mqsichangeflowuserexits command.

    mqsichangeflowuserexits <integrationNodeName> -o -x /var/mqsi/shared-classes

    If you extract IBM ACE tracing .tar files into /opt/acetracingexit directory, replace /var/mqsi/shared-classes with /opt/acetracingexit.

  3. Activate the user exit.

    User exits can be active or inactive, and are inactive by default. You can activate user exit for an integration node, an integration server, or a specific message flow.

    1. Activate the user exit for an integration node.

      1. Activate the user exit:

        mqsichangeflowuserexits <integrationNodeName> -o -a ACEOpenTracingUserExit

      2. Verify the user exit:

        mqsireportflowuserexits <integrationNodeName> -o

        See the following sample output:

        # mqsireportflowuserexits BK3 -o
BIP8854I: User Exits active for integration server 'BK3': ACEOpenTracingUserExit.
BIP8855I: User Exits inactive for integration server 'BK3': .
BIP8741I: User Exit path for integration server 'BK3': /var/mqsi/shared-classes.
BIP8071I: Successful command completion.

      3. Start the integration node:

        mqsistart <integrationNodeName>

    2. Activate the user exit for an integration server.

      1. Start the integration node:

        mqsistart <integrationNodeName>

      2. Activate the user exit:

        mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ACEOpenTracingUserExit

    3. Activate the user exit for a message flow.

      1. Start the integration node:

        mqsistart <integrationNodeName>

      2. Activate the user exit for a message flow:

        mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ACEOpenTracingUserExit

    4. Repeat the steps for other integration nodes, integration servers, or message flows for which you want to activate the user exit.

For more information, see the following links:

Configuring Instana ACE Tracing

  1. Go to the /var/mqsi/shared-classes directory.

  2. Edit the acetracingexit.conf file:

    # configuration for ace tracing exit
LOG_LEVEL="info" #Log level: info, warn, error, debug
SPAN_FORMAT="instana"
CICS_SUPPORT="off" #Propagate trace context for CICS request: off, on
MONITOR_LEVEL="verbose" #ACE tracing level: off, normal, verbose
INSTANA_AGENT_HOST="localhost" #(optional)
INSTANA_AGENT_PROTO="http" #(optional)
HOST_ALIAS="<YOUR-HOST-NAME>" #(optional)

    where:

    • LOG_LEVEL
      Specifies the log level, which can be one of the following types: info, warn, error, or debug. The log file is at the /tmp/trace directory.
    • SPAN_FORMAT
      Specifies where the span data is sent to. Set this variable to instana. Instana ACE Tracing user exit sends span data to the host agent endpoint http://localhost:42699 by default. Update the configuration fields INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO if you want to send the span data to a remote host agent that uses HTTPS protocol. You need to have the same SPAN_FORMAT setting for all IBM ACE instances.
    • CICS_SUPPORT This parameter controls whether the support for tracing CICS request is enabled. Set it to on to enable and off to disable the support.
    • MONITOR_LEVEL
      Specifies the tracing level of IBM ACE, which can be one of the following types: off, normal, or verbose. If MONITOR_LEVEL is set to off, no tracing context is appended to the outgoing request. If MONITOR_LEVEL is set to normal, the tracing context is appended only when the IBM MQ message includes the RFH2 header. If MONITOR_LEVEL is set to verbose, the tracing context is appended to all outgoing HTTP or IBM MQ requests.
    • INSTANA_AGENT_HOST
      Specifies the agent host where the Instana format span data is sent to. By default, localhost is used. If you specify a remote agent host, you must also add a line http.listen=* in <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
    • INSTANA_AGENT_PROTO
      Specifies the connection type between IBM ACE Tracing user exit and the host agent. By default, http is used. However, https is also supported. If you want to change it to https, you need to follow Set up TLS Encryption for Agent Endpoint to secure the Instana agent endpoint first.
    • HOST_ALIAS
      Specifies a host alias for the span data that is collected by Instana ACE Tracing user exit. So calls to IBM ACE can be linked to the infrastructure entity if the integration node or integration server is also monitored by the IBM ACE sensor. The FQDN of the IBM ACE host is used by default. The host alias value needs to match with the IBM ACE sensor host that is specified in the host agent configuration yaml file. You need to specify a host alias only if the FQDN of the IBM ACE host is not used in the Instana ACE sensor configuration when the host agent is not on the local IBM ACE host. The IBM ACE sensor can discover the FQDN for the local integration nodes or integration servers.

  3. Save the file and restart integration node or integration server.

Repeat these installation and configuration steps on other IBM ACE hosts that you want to enable tracing for.

You can view Instana ACE Tracing data in the Instana UI.

Unconfiguring Instana ACE Tracing user exit

  1. Unconfigure user exit.

    1. Unconfigure user exit for an integration node:

      1. Stop the integration node:

        mqsistop <integrationNodeName>

      2. Deactivate the user exit:

        mqsichangeflowuserexits <integrationNodeName> -o -a ""

      3. Restart the integration nodes.

    2. Deactivate user exit for an integration server:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ""

    3. Deactivate user exit for a message flow:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ""

  2. Repeat steps on other integration nodes.

Enabling tracing for ACE 12.0.7 or later in the traditional ACE environment

From ACE 12.0.7.0, the native tracing support based on OpenTelemetry is added, and the tracing calls can be accepted by Instana agent's OpenTelemetry ingestion. You don't need to install an extra Instana ACE Tracing user exit for ACE. You must use ACE native tracing if the ACE version is greater than or equal to 12.0.7.0. Do not enable Instana ACE Tracing user exit and ACE native tracing simultaneously because the tracing calls mix with the upstream or downstream tracing calls.

Integration servers support OpenTelemetry trace on the following platforms:

  • AIX (IBM ACE 12.0.10.0 and later)
  • Linux x86-64 (IBM ACE 12.0.7.0 and later)
  • Linux on System z (IBM ACE 12.0.8.0 and later)
  • Linux on Power Systems - Little Endian (IBM ACE 12.0.10.0 and later)
  • Windows (IBM ACE 12.0.8.0 and later)

Configuring ACE native tracing

To enable native tracing for IBM ACE, follow the steps:

  1. Configure OpenTelemetry data ingestion for Instana. For more information, see Configuring OpenTelemetry data ingestion.

  2. Enable ACE native tracing. For more information, see Configuring OpenTelemetry trace for an integration server.

For more information about considerations and limitations, see OpenTelemetry considerations and limitations.

Enabling tracing in the IBM Cloud Pak for Integration environment

You can use Instana AutoTrace WebHook to enable tracing for ACE integration server in the IBM Cloud Pak for Integration (amd64 only). Depending on the following ACE versions, the Instana AutoTrace WebHook deploys different tracing technologies in the ACE integration server instance:

  • For ACE versions before 12.0.8, the Instana user exit for ACE Tracing is deployed automatically into the ACE integration server instance.
  • For ACE 12.0.8 or later, the native OpenTelemetry Tracing for ACE is automatically enabled for the ACE integration server instance.

The OpenTelemetry data ingestion for Instana must be enabled. For more information, see Configuring OpenTelemetry data ingestion.

After the ACE Tracing is enabled, click Applications > Services in the Instana UI to check the tracing call details. The service name for the ACE Tracing that is enabled through these options are as follows:

  • Instana ACE user exit: The service name includes Pod IP and integration server name of ACE, and the format is <ACE_Pod_IP>-<IntegrationServer_Name>:<IntegrationServer_Name>, for example, 10.254.17.125-is-01-1206:is-01-1206.
  • ACE native OpenTelemetry: The service name includes the integration server name, and the format is IBM App Connect Enterprise-<IntegrationServer_Name>, for example, IBM App Connect Enterprise-is-01-customer.

For more information about how to enable tracing for container ACE that runs in IBM Cloud Pak for Integration, see Instana AutoTrace WebHook: IBM MQ and ACE.

Troubleshooting Instana ACE Tracing

You might encounter some issues with Instana ACE Tracing. For more information about how to resolve them, see Troubleshooting.