Monitoring IBM App Connect Enterprise (ACE)

After you install the Instana host agent, IBM ACE sensor is automatically installed, but you need to configure IBM ACE sensor as outlined in this topic. Then, you can view metrics that are related to IBM ACE in the Instana UI.

Supported information

Supported operating systems

The supported operating systems of IBM ACE sensor are consistent with host agents requirements, which can be checked in the Supported operating systems section of each host agent, such as Supported operating systems for Unix.

Supported ACE versions and platforms

IBM ACE sensor and Instana ACE tracing have different version and platform requirements. For more information, see Supported ACE versions and platforms.

Notes:

  • For IBM Integration Bus v10, make sure that 10.0.0.24 or later version is used. Or else, the "Number of Thread In Pool" does not report the correct value in accordance with the WLM policy.
  • In cloud-native environment, only IBM App Connect Enterprise certified container image is supported.
    Other supported information
  • IBM ACE sensor supports both local monitoring and remote monitoring.

Configuring

You can configure IBM ACE sensor in a traditional or cloud-native environment.

Prerequisites

  • If the server is IBM Integration Bus v10, use the following commands to check the statistics state. Turn on or turn off statistics collection when needed.
    Note: The metrics return only when the statistics gathering and resource statistics collection are active.

    • Check resource statistics state and turn on if needed. See the following example:
    # Check the collection state of resource statistics for specified server in specified integration node.
mqsireportresourcestats integrationNode -e integrationServer
# If the state is inactive, turn on the resource statistics collection for specified integration server in specified integration node.
mqsichangeresourcestats integratorNode -e integrationServer -c active

    For more information, see mqsireportresourcestats and mqsichangeresourcestats.

    • Check statistics state and turn on if needed. See the following example:
    # Check the statistics gathering state for all message flows in sepcified integration server for specified integrator node. "-s" means snapshot type
mqsireportflowstats integratorNode  -s -e integrationServer  -j
# If the state is inactive, turn on the collection of snapshot statistics data for all message flows in specified integration server for specified integration node, and emit the data in JSON format.
# Example-1 without setting thread data level:
mqsichangeflowstats integrationNode -s -e integrationServer -j -c active -o json
# Example-2 with setting thread data level:
mqsichangeflowstats integrationNode -s -e integrationServer -j -t basic -c active -o json
 # Example-3 with setting both thread data level and node data level:
mqsichangeflowstats integrationNode -s -e integrationServer -j -t basic -n basic -c active -o json

    For more information, see mqsireportflowstats and mqsichangeflowstats.

    Notes:

    • If you want to check the thread use of each message flow, you need to enable thread data with option -t basic when you are using the mqsichangeflowstats command.
    • You can request two types of data collection, snapshot and archive. For more information, see Message flow accounting and statistics collection options. It is good to use snapshot type for IBM ACE Sensor.

  • If IBM MQ is used for ACE/IIB traditional environment, set up a channel. If the integration node or server is associated with a queue manager, you must set up a server connection channel. The performance metrics data for integration server, message flow, and flow node flows through this channel. This channel is accessible by the username or password that is specified in the configuration.yaml file. In addition to the channel name, username, and password, you also need to specify the queue manager name and listening port in configuration.yaml. If you have any issue about channel configuration, refer to Resolving CHLAUTH access issues.

Configuring for traditional ACE

For traditional ACE, Instana supports monitoring of both remote and local IBM ACE instances. You need to configure the following fields in the agent configuration <agent_install_dir>/etc/instana/configuration.yaml file:

  • To use remote monitoring, set the IP address of target ACE server as value of host. Or else, comment out the host attribute and local monitoring is used by default.

  • To monitor brokers or integration servers with the same name on different hosts, use brokerName:x or integrationServer:x as the key of the brokers or integration servers, such as BK1:1and BK:2.

To get tips about how to get the values for these fields, see the Useful tips for ACE sensor configuration blog.

See the following sample configuration.yaml file:

com.instana.plugin.ace:
  enabled: true
  poll_rate: 60                  # Default value is 60 seconds, and the minimum value is 20 seconds.
  flowNodesExcludedRegex: ''     # Global regex for filtering exclusive flow nodes from all integration nodes (optional)
  NodesOrServers:                # Multiple Integration node instances or multiple standalone Integration servers can be specified
    BK1:  # Example1: Integration Node with MQTT (required)
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      ######################################################################################################
      # host: 'aceserver.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'            # ACE server's rest api port (required)
      #aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      #acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      #keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      #keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      #excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      #flowNodesExcludedRegex: ''    # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      #availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      # Note: if MQTT is used, then keep mqport only and comment out other MQ parameters.
      #######################################################################################################
      #mqHost: 'mqserver.com'        # MQ host/IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '11883'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      #queuemanagerName: 'QM1'       # Queue Manager name (required for IBM MQ)
      #channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      #mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      #mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
    IS1:  # Example2: High Availability integration Node with MQ configured. (required)
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      ######################################################################################################
      host: 'ha-instance1.com,ha-instance2.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'            # ACE server's rest api port (required)
      #aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      #acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      #keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      #keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      #excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      #flowNodesExcludedRegex: ''    # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      #availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      #######################################################################################################
      #mqHost: 'mqserver.com'       # MQ host or IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '1414'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM2'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
    BK1:2:  # Example3: Same broker names on different hosts, such as BK1, BK1:1, BK1:2, BK1:n, etc.
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the \u2018#\u2019 when needed.
      ######################################################################################################
      host: 'aceserver.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'           # ACE server's rest api port (required)
      aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      # keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      # keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      # excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      # flowNodesExcludedRegex: ''  # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      # availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the \u2018#\u2019 when needed.
      # Note: if MQTT is used, then keep mqport only and comment out other MQ parameters.
      #######################################################################################################
      mqHost: 'mqserver.com'        # MQ host/IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '1414'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM1'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)

Configuring for cloud-native ACE

Configuration for ACE sensor in the cloud-native environment is auto-discovered.

Viewing metrics

To view the metrics, complete the following steps:

  1. In the sidebar of the Instana UI, select Infrastructure.
  2. Click a specific monitored host.

You can see a host dashboard with all the collected metrics and monitored processes.

Configuration data

  • Default Queue Manager Name
  • Product Name
  • Version
  • Build Level
  • Platform Name
  • Platform Architecture
  • Platform Version
  • Application Name

Performance metrics

Integration server

Metric Description
Initial Heap Memory The initial amount of memory that the JVM requests from the operating system for memory management during startup.
Used Heap Memory The amount of memory that is in use.
Committed Heap Memory The amount of memory that is allocated to the JVM by the operating system.
Max Heap Memory The maximum amount of memory that can be used for memory management.
Initial Nonheap Memory The initial amount of memory that the JVM requests from the operating system for memory management during startup.
Used Nonheap Memory The amount of memory that is in use.
Committed Nonheap Memory The amount of memory that is allocated to the JVM by the operating system.
Max Nonheap Memory The maximum amount of memory that can be used for memory management.
Cumulative GC Time The time when the data is collected, in seconds.
Cumulative Number of GC The total number of garbage collections occurred for this instance of the JVM.

Message flow

Metric Data type Description
Total Elapsed Time Numeric Total elapsed time that a message flow in microseconds spent processing input messages.
Maximum Elapsed Time Numeric Maximum elapsed time that a message flow in microseconds spent processing an input message.
Minimum Elapsed Time Numeric Minimum elapsed time that a message flow in microseconds spent processing an input message.
Total CPU Time Numeric Total CPU time that a message flow in microseconds spent processing an input message.
Max CPU Time Numeric Maximum CPU time that a message flow in microseconds spent processing an input message.
Min CPU Time Numeric Minimum CPU time that a message flow in microseconds spent processing an input message.
CPU Time Waiting For Input messages Numeric Total CPU time that a message flow in microseconds spent waiting for input messages.
Elapsed Time Waiting For Input messages Numeric Total elapsed time in microseconds spent waiting for input messages.
Num of All Input messages Numeric Total number of input messages.
Total Size of Input Messages Numeric Total size of input messages in bytes.
Maximum Size of Input Messages Numeric Maximum size of input messages in bytes.
Minimum Size of Input Messages Numeric Minimum size of input messages in bytes.
Number of Threads in Pool Numeric Number of threads in pool.
Number of Times That Max Threads Reached Numeric Number of times the maximum number of threads is reached.
Number of MQGET Errors Numeric Number of MQGET errors for MQInput nodes or Web Services errors for HTTPInput nodes.
Number of Messages with Errors Numeric Number of messages that contain errors.
Number of Errors when you are processing Messages Numeric Number of errors that occur when you are processing a message.
Number of Transaction timeouts Numeric Number of transaction timeouts that occur when you are processing a message (for AggregateReply nodes only).
Number of Transaction Commits Numeric Number of transaction commits that occur when you are processing a message.
Number of Transaction Backouts Numeric Number of transaction backouts that occur when you are processing a message.
Thread Utilization Numeric Number of thread utilizations for each message flow.

Message flow node

Metric Data type Description
Total Elapsed Time Numeric Total elapsed time in microseconds spent processing input messages.
Max Elapsed Time Numeric Maximum elapsed time in microseconds spent processing input messages.
Min Elapsed Time Numeric Minimum elapsed time in microseconds spent processing input messages.
Total CPU Time Numeric Total cpu time in microseconds spent processing input messages.
Max CPU Time Numeric Maximum CPU time in microseconds spent processing an input message.
Min CPU Time Numeric Minimum CPU time in microseconds spent processing an input message.
Number of Invocations Numeric Total number of messages processed by this flow node.
Number of Input-type Terminals Numeric Number of input-type terminals.
Number of Output-type Terminals Numeric Number of output-type terminals.

Known Limitations

  • If HTTPS is enabled for REST API interface, you need to specify keystore and keystorePassword parameters in <agent_install_dir>/etc/instana/configuration.yaml. For keystore type, only JKS or P12 is supported.

Health Signatures

Each sensor has a curated knowledgebase of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents, which depend on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any entity.

For more information about built-in events for IBM ACE sensor, see Built-in events reference.

Troubleshooting

  • Test the IBM MQ channel connectivity
    If you want to test the IBM MQ channel connectivity, you must set up the IBM MQ Explorer first. Then, you can test the channel connectivity in the IBM MQ Explorer with the credential that is provided in the configuration.yaml file.

  • Resolve warning message due to automatic discovery failure
    If the automatic discovery fails, you might get a warning message on your dashboard UI. The explanation and solution for these warning messages are as follows:

    • Auto-discovery: The OS Windows 10 is not supported for automatic discovery.
      This message is displayed if you have OS Windows 10 running on the host. OS Windows 10 is not supported for automatic discovery. To resolve this issue, you must specify integration node information in the <agent_install_dir>/etc/instana/configuration.yaml file to continue monitoring with rest API.
    • Auto-discovery: The OS Windows 10 or the ACE/IIB v10 is not supported for automatic discovery currently.
      This message is displayed when the host either has OS Windows 10 or IBM Integration Bus v10 running. Both of them are not supported. To resolve this issue, follow the same solution as outlined in the previous message.
    • Auto-discovery: Unix Domain Socket is unable to access as the mqsi work path is null.
      This message is displayed when IBM ACE sensor fails to get the WebSphere Message Broker work path or when the WebSphere Message Broker path is null. To resolve this issue, follow the same solution as outlined in the first message.
    • Auto-discovery: MQTT is disabled. You can enable MQTT or specify MQ information in the configuration.yaml file.
      This message is displayed when the MQTT process is disabled or stopped unexpectedly. To resolve this issue, enable MQTT and start the process. If you don't want to enable it, refer to the same solution as outlined in the first message.

  • Fix the error code 2058 displayed on connecting to queue manager
    If you cannot get metrics from the Instana UI but instead get an error code 2058 from the agent log, ensure that all IBM MQ related information is correct in the configuration file. Run the ps -ef | grep runmqlsr | grep '<QUEUEMGR NAME>' | grep '<LISTENER_PORT>' command to list the detailed information. If the listener has a specified IP with -i, you must remove the limitation or set up a new listener for ACE.
    See the following example output with correct setting:

    root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1414
mqm       9956  9872  0 Aug30 ?        00:00:05 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1414

    See the following example output with wrong setting:

    root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1417
mqm       9957 9872  0 Aug30 ?        00:00:06 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1417 -i 10.17.59.113

  • Obtain the REST API port of the current integration node
    If you want to get the REST API port that the current integration node is using, run the following command:

    # If HTTP protocal is used, get the port with this command
mqsireportproperties integrationNode -b webadmin -o HTTPConnector -a | grep port
# If HTTPS protocal is used, get the port with this command
mqsireportproperties integrationNode  -b webadmin -o HTTPSConnector -a | grep port

  • Collect the payload in JSON format
    If you want to get the payload in JSON format for further investigation, you can use the following command:

    # CMD for IIBv10
curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv1?depth=6" >> payload.json
# CMD for ACEv11
curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv2?depth=6" >> payload.json

  • Remove the warning message Monitoring issue: jvm_attach_generic from the integration server dashboard
    This message is displayed when the Instana agent can't correctly attach the DataFlowEngine process, which is a JVM wrapper.

    • If you are using IBM Integration Bus v10, you can ignore this process in <agent_install_dir>/etc/instana/configuration.yaml as follows:
       com.instana.ignore:
   processes:
     - 'DataFlowEngine'

    For more information, see Ignore processes.
    Note: The potential impacts of ignoring this process are as follows:
    * The host process of integration server is set to bipservice by default. * The CPU load or usage is improved by ignoring these processes especially for a large-scale environment, which might have many integration servers that are running with the DataFlowEngine process.

    • If you are using IBM App Connect Enterprise v11 or v12, you can choose to ignore this process with the steps as described or fix the warning by completing the following steps:

      1. Set up the JAVA_HOME environment variable, and point it to the ACE bundled JRE by running the following command:
      export JAVA_HOME= <PATH to your ACE Bundled JRE>

      See the following example:

      export JAVA_HOME=/opt/ace-11.0.0.11/common/jdk/jre
      1. Stop and start the ACE broker (mqsistop/mqsistart).

  • Restore missing flow nodes table and the related metrics for self-hosted environment
    A design change was introduced for flow node metrics since release-217. The flow nodes dashboard page is removed, and all flow nodes metrics are displayed with expandable tables. For self-hosted environments, you need to choose one of following solutions to see flow node metrics again:

    • Upgrade the self-hosted version to release-217. With this solution, you can get all the new features of Instana.
    • If you don't want to upgrade the backend to use the features of the old agent, you can pin the agent to version 1.1.620. For more information, see version pinning.
      Note: If you pin the agent to version 1.1.620, all sensors will be on old versions.

  • Resolve the error message Create WMQ connection for ACE sensor failed on the Windows Instana agent
    If you don't get the metrics of the Windows agent from the Instana UI, but instead get an error message Create WMQ connection for ACE sensor failed on the agent UI, ensure that the IBM MQ security feature is disabled. If you still get the error message after IBM MQ security is disabled, check whether the IBM MQ user is added to the Windows user group mqm.

Tracing

Platforms

The Instana ACE tracing user exit supports:

  • IBM App Connect Enterprise v11.0.0.8 or later.
  • Traditional ACE in Linux x86_64, Linux ppc64le, Linux s390x, AIX 7.2, AIX 7.3, Windows, and container ACE in IBM Cloud Pak for Integration (amd64 only).

Notes:

Prerequisites

Supported node types

The Instana ACE tracing user exit supports HTTP request, IBM MQ request, and Kafka request only. IBM MQ messages that include only an MQRFH2 header are supported because the trace information is written into IBM MQ message's MQRFH2 header to propagate the trace context.

IBM MQ message requirements

In some IBM MQ consumer clients, the presence of additional header data in IBM MQ messages can cause message processing errors and message rejection.

Important: You must test IBM MQ consumer clients in a nonproduction environment before you enable it in a production environment. If on enabling trace correlation support causes errors in IBM MQ client applications, do one of the following actions:

  • If it is an application that can be changed, update the application to ignore the additional IBM MQ header data that is added by Instana. You can contact IBM MQ support if assistance is needed to make IBM MQ client changes.
  • If it is an application that cannot be changed, do not enable IBM ACE trace correlation support.

Others

The Instana ACE tracing user exit supports only message flows with input node as the entrypoint.

IBM ACE must be installed and configured before you install and configure this component.

Enabling tracing for ACE version earlier than v12.0.7 in traditional ACE environment

Downloading IBM ACE Tracing user exit

  1. Download the IBM ACE Tracing user exit .tgz file from the artifactory. To download the file, use _ as the username and a valid agent key as the password.

  2. Extract the downloaded .tgz file to a temporary location. After extraction, find five user-exit packages for different platforms in the directory.

  3. Transfer the user exit package specific to a platform on your IBM ACE host.

  4. Extract the user exit package into the /var/mqsi/shared-classes directory on your ACE host. Place the following files in your shared classes directory:

    • ACEOpenTracingUserExit.lel: This file contains the Instana ACE user exit, which intercepts the HTTP request, IBM MQ request, and Kafka request, and starts the wrapped OpenTelemetry C++ client library to create spans.
    • tracelibrary.so: This file specifies the wrapped OpenTelemetry C++ client, which provides functions to manage the lifecycle of spans, and sends spans to the target-tracing system.
    • acetracingexit.conf: This configuration file specifies the log level and information about connecting to the host agent.

    Note: If the ACE server is not installed with global installation, the /var/mqsi/shared-classes directory does not exist on your IBM ACE host. Create the directory /opt/acetracingexit manually on your IBM ACE host, and extract the tar file into the directory.

Configuring user exit

To enable tracing for IBM ACE, complete the following steps:

  1. Stop the integration node.

    mqsistop <integrationNodeName>

  2. Install the user exit on an integration node by setting the UserExitPath property that uses the mqsichangeflowuserexits command.

    mqsichangeflowuserexits <integrationNodeName> -o -x /var/mqsi/shared-classes

    Note: If you extract IBM ACE tracing tar files into /opt/acetracingexit directory, replace /var/mqsi/shared-classes with /opt/acetracingexit.

  3. Activate the user exit.

    User exits can be active or inactive, and are inactive by default. You can activate user exit for an integration node, an integration server, or a specific message flow.

    1. Activate the user exit for an integration node.

      1. Activate the user exit:

        mqsichangeflowuserexits <integrationNodeName> -o -a ACEOpenTracingUserExit

      2. Verify the user exit:

        mqsireportflowuserexits <integrationNodeName> -o

        See the following example output:

        # mqsireportflowuserexits BK3 -o
BIP8854I: User Exits active for integration server 'BK3': ACEOpenTracingUserExit.
BIP8855I: User Exits inactive for integration server 'BK3': .
BIP8741I: User Exit path for integration server 'BK3': /var/mqsi/shared-classes.
BIP8071I: Successful command completion.

      3. Start the integration node:

        mqsistart <integrationNodeName>

    2. Activate the user exit for an integration server.

      1. Start the integration node:

        mqsistart <integrationNodeName>

      2. Activate the user exit:

        mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ACEOpenTracingUserExit

    3. Activate the user exit for a message flow.

      1. Start the integration node:

        mqsistart <integrationNodeName>

      2. Activate the user exit for a message flow:

        mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ACEOpenTracingUserExit

    4. Repeat the steps for other integration nodes, integration servers, or message flows for which you want to activate the user exit.

For more information, see the following links:

Configuring Instana ACE Tracing

  1. Go to the /var/mqsi/shared-classes directory.

  2. Edit the acetracingexit.conf file:

    # configuration for ace tracing exit
LOG_LEVEL="info" #Log level: info, warn, error, debug
SPAN_FORMAT="instana"
MONITOR_LEVEL="verbose" #ACE tracing level: off, normal, verbose
INSTANA_AGENT_HOST="localhost" #(optional)
INSTANA_AGENT_PROTO="http" #(optional)
HOST_ALIAS="<YOUR-HOST-NAME>" #(optional)

    where:

    • LOG_LEVEL
      Specifies the log level, which can be one of the following types: info, warn, error, or debug. The log file is at the /tmp/trace directory.
    • SPAN_FORMAT
      Specifies where the span data is sent to. Set this variable to instana. Instana ACE Tracing user exit sends span data to the host agent endpoint http://localhost:42699 by default. Update the configuration fields INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO if you want to send the span data to a remote host agent that uses HTTPS protocol. You need to have the same SPAN_FORMAT setting for all IBM ACE instances.
    • MONITOR_LEVEL
      Specifies the tracing level of IBM ACE, which can be one of the following types: off, normal, or verbose. If MONITOR_LEVEL is set to off, no tracing context is appended to outgoing request. If MONITOR_LEVEL is set to normal, the tracing context is appended only when the IBM MQ message includes the RFH2 header. If MONITOR_LEVEL is set to verbose, the tracing context is appended to all outgoing HTTP/MQ requests.
    • INSTANA_AGENT_HOST
      Specifies the agent host where the Instana format span data is sent to. By default, localhost is used. If you specify a remote agent host, you must also add a line http.listen=* in <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
    • INSTANA_AGENT_PROTO
      Specifies the connection type between IBM ACE tracing user exit and the host agent. By default, http is used. However, https is also supported. If you want to change it to https, you need to follow Setup TLS Encryption for Agent Endpoint to secure the Instana agent endpoint first.
    • HOST_ALIAS
      Specifies a host alias for the span data that is collected by Instana ACE Tracing user exit. So calls to IBM ACE can be linked to the infrastructure entity if the integration node or integration server is also monitored by IBM ACE sensor. The FQDN of IBM ACE host is used by default. The host alias value needs to match with IBM ACE sensor host that is specified in host agent configuration yaml. You need to specify a host alias only if the FQDN of IBM ACE host is not used in Instana ACE sensor configuration when the host agent is not on the local IBM ACE host. IBM ACE sensor can discover the FQDN for the local integration nodes or integration servers.

  3. Save the file and restart integration node or integration server.

Repeat these installation and configuration steps on other IBM ACE hosts that you want to enable tracing for.

You can view Instana ACE tracing data in the Instana UI.

Unconfiguring Instana ACE Tracing user exit

  1. Unconfigure user exit.

    1. Unconfigure user exit for an integration node:

      1. Stop the integration node:

        mqsistop <integrationNodeName>

      2. Deactivate the user exit:

        mqsichangeflowuserexits <integrationNodeName> -o -a ""

      3. Restart the integration nodes.

    2. Deactivate user exit for an integration server:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ""

    3. Deactivate user exit for a message flow:

      mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ""

  2. Repeat steps on other integration nodes.

Enabling tracing when ACE version equal or greater than v12.0.7 in traditional ACE environment

From App Connect Enterprise 12.0.7.0, the native tracing support based on OpenTelemetry is added, and the tracing calls can be accepted by Instana agent's OpenTelemetry ingestion. You don't need to install additional Instana ACE tracing user exit for ACE. You must use ACE native tracing if the ACE version is greater than or equal to 12.0.7.0. Do not enable Instana ACE tracing user exit and ACE native tracing simultaneously because the tracing calls mix with the upstream or downstream tracing calls.

Configuring ACE native tracing

To enable native tracing for IBM ACE, follow the steps:

  1. Activate OpenTelemetry Support for Instana agent. For more information, see Activating OpenTelemetry Support.

  2. Enable ACE native tracing. For more information, see Configuring OpenTelemetry trace for an integration server.

For more information about considerations and limitations, see OpenTelemetry considerations and limitations.

Enabling tracing in CP4I environment

You can use Instana AutoTrace WebHook to enable Instana ACE Tracing in IBM Cloud Pak for Integration (amd64 only). You don't need to install the Instana ACE tracing user exits manually in the Cloud Pak for Integration environment. For more information about how to enable tracing for container ACE that runs in Cloud Pak for Integration, see Instana AutoTrace WebHook - IBM MQ and ACE.

For more information, see the Instana AutoTrace WebHook documentation

Troubleshooting Instana ACE tracing

  • Check whether Instana ACE tracing is enabled
    To verify whether the Instana ACE tracing is enabled or not, follow the steps:

    1. Check the integration server's stdout file, such as /var/mqsi/components/BK1/servers/EG1/stdout. If the Instana ACE tracing is enabled successfully, you can see the following message:
      Load Tracing Agent in shared mode, 0x7f322822bbbe, (nil)
/var/mqsi/shared-classes/tracelibrary.so
Found agent library!
    2. Check whether the /tmp/trace directory is generated or not. If the Instana ACE tracing is enabled successfully, the user exit creates the log file like aceExit.13312.log in the /tmp/trace directory. The number 13312 is the DataFlowEngine or Integration Server process ID. See the following example:
      [root@hastened1 trace]# ps -ef | grep 13312
root     13312 13260 13 18:37 ?        00:00:29 DataFlowEngine v12_BK1 00000000-0000-0000-0000-000000000000 BK1-EG1
    3. If Instana ACE tracing fails to be enabled, collect the system log /var/log/message, the integration node's stdout and stderr file, and the integration server's stdout and stderr file. Send all files to Instana support team.

  • Missing ACE tracing calls in the Instana UI If you can't see the ACE tracing calls in the Instana UI or tracing call doesn't show as expected, follow the steps:

    1. Enable debugging for Instana ACE tracing.
    2. Restart the Integration Server, and collect all files in the /tmp/trace directory. Collect the tracing call's screen captures on the Instana UI. Send the debug files and screen captures to the Instana Support team.

Automatic discovery in ACE

Automatic discovery feature in ACE sensor discovers all integration servers that are running on the host and starts monitoring them automatically without any configuration.

Supported versions

  • Linux and AIX are supported. Windows is not supported.
  • IBM App Connect Enterprise v11 or later is supported. IBM Integration Bus v10 is not supported.

Prerequisites

Ensure that the following requirements are met for the automatic discovery feature in ACE:

  • ACE sensor is enabled without configuring integration node or server in the <agent_install_dir>/etc/instana/configuration.yaml file.
  • Curl is installed on the host. To check the curl installation, run the which curl command.
  • Permission is available to access WebSphere Message Broker work path (for example, /var/mqsi/) to start the agent.
  • MQTT process is running properly for related integration node. To check, run the ps -ef | grep -E 'bipMQTT.*<INTEGRATION_NODE>’ command.

Note: With automatic discovery, you can see all running integration servers that are listed on the host dashboard of the Instana UI even when all the prerequisites are not met.