Monitoring IBM App Connect Enterprise (ACE)

Supported operating systems

The supported operating systems of IBM ACE sensor are consistent with host agents requirements, which can be checked in the Supported operating systems section of each host agent, such as Supported operating systems for Unix.

Supported ACE versions and platforms

IBM ACE sensor and Instana ACE tracing have different version and platform requirements. For more information, see Supported ACE versions and platforms.

Notes:

• For IBM Integration Bus v10, make sure that 10.0.0.24 or later version is used. Or else, the "Number of Thread In Pool" does not report the correct value in accordance with the WLM policy.
• In cloud-native environment, only IBM App Connect Enterprise certified container image is supported.
  Other supported information
• IBM ACE sensor supports both local monitoring and remote monitoring.

Prerequisites

• If the server is IBM Integration Bus v10, use the following commands to check the statistics state. Turn on or turn off statistics collection when needed.
  Note: The metrics return only when the statistics gathering and resource statistics collection are active.

  • Check resource statistics state and turn on if needed. See the following example:

  # Check the collection state of resource statistics for specified server in specified integration node.
  mqsireportresourcestats integrationNode -e integrationServer
  # If the state is inactive, turn on the resource statistics collection for specified integration server in specified integration node.
  mqsichangeresourcestats integratorNode -e integrationServer -c active
  

  For more information, see mqsireportresourcestats and mqsichangeresourcestats.

  • Check statistics state and turn on if needed. See the following example:

  # Check the statistics gathering state for all message flows in sepcified integration server for specified integrator node. "-s" means snapshot type
  mqsireportflowstats integratorNode  -s -e integrationServer  -j
  # If the state is inactive, turn on the collection of snapshot statistics data for all message flows in specified integration server for specified integration node, and emit the data in JSON format.
  # Example-1 without setting thread data level:
  mqsichangeflowstats integrationNode -s -e integrationServer -j -c active -o json
  # Example-2 with setting thread data level:
  mqsichangeflowstats integrationNode -s -e integrationServer -j -t basic -c active -o json
   # Example-3 with setting both thread data level and node data level:
  mqsichangeflowstats integrationNode -s -e integrationServer -j -t basic -n basic -c active -o json
  

  For more information, see mqsireportflowstats and mqsichangeflowstats.

  Notes:

  • If you want to check the thread use of each message flow, you need to enable thread data with option -t basic when you are using the mqsichangeflowstats command.
  • You can request two types of data collection, snapshot and archive. For more information, see Message flow accounting and statistics collection options. It is good to use snapshot type for IBM ACE Sensor.

• If IBM MQ is used for ACE/IIB traditional environment, set up a channel. If the integration node or server is associated with a queue manager, you must set up a server connection channel. The performance metrics data for integration server, message flow, and flow node flows through this channel. This channel is accessible by the username or password that is specified in the configuration.yaml file. In addition to the channel name, username, and password, you also need to specify the queue manager name and listening port in configuration.yaml. If you have any issue about channel configuration, refer to Resolving CHLAUTH access issues.

Configuring for traditional ACE

For traditional ACE, Instana supports monitoring of both remote and local IBM ACE instances. You need to configure the following fields in the agent configuration <agent_install_dir>/etc/instana/configuration.yaml file:

• To use remote monitoring, set the IP address of target ACE server as value of host. Or else, comment out the host attribute and local monitoring is used by default.

• To monitor brokers or integration servers with the same name on different hosts, use brokerName:x or integrationServer:x as the key of the brokers or integration servers, such as BK1:1and BK:2.

To get tips about how to get the values for these fields, see the Useful tips for ACE sensor configuration blog.

See the following sample configuration.yaml file:

com.instana.plugin.ace:
  enabled: true
  poll_rate: 60                  # Default value is 60 seconds, and the minimum value is 20 seconds.
  flowNodesExcludedRegex: ''     # Global regex for filtering exclusive flow nodes from all integration nodes (optional)
  NodesOrServers:                # Multiple Integration node instances or multiple standalone Integration servers can be specified
    BK1:  # Example1: Integration Node with MQTT (required)
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      ######################################################################################################
      # host: 'aceserver.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'            # ACE server's rest api port (required)
      #aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      #acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      #keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      #keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      #excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      #flowNodesExcludedRegex: ''    # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      #availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      # Note: if MQTT is used, then keep mqport only and comment out other MQ parameters.
      #######################################################################################################
      #mqHost: 'mqserver.com'        # MQ host/IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '11883'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      #queuemanagerName: 'QM1'       # Queue Manager name (required for IBM MQ)
      #channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      #mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      #mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
    IS1:  # Example2: High Availability integration Node with MQ configured. (required)
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      ######################################################################################################
      host: 'ha-instance1.com,ha-instance2.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'            # ACE server's rest api port (required)
      #aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      #acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      #keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      #keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      #excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      #flowNodesExcludedRegex: ''    # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      #availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the ‘#’ when needed.
      #######################################################################################################
      #mqHost: 'mqserver.com'       # MQ host or IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '1414'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM2'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
    BK1:2:  # Example3: Same broker names on different hosts, such as BK1, BK1:1, BK1:2, BK1:n, etc.
      ######################################################################################################
      # Following Parameters are used to access ACE server's rest api service to retrieve status data, configuration data, etc.
      # Optional parameters are commented out by default. Remove the \u2018#\u2019 when needed.
      ######################################################################################################
      host: 'aceserver.com'  # Host of ACE server or hosts of HA with multi-instance/RDQM system for remote monitoring. For HA, seperate multiple hosts by comma. (optional)
      restApiPort: '4414'           # ACE server's rest api port (required)
      aceUsername: 'viewer'         # ACE rest api service username if security is enabled (optional)
      acePassword: 'viewerPassword' # ACE rest api service password if security is enabled (optional)
      # keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      # keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      # excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring. You can separate multiple names by comma (optional)
      # flowNodesExcludedRegex: ''  # Regex for filtering exclusive flow nodes from the integration node. The regex that is set in this property has higher priority than the global regex. (optional)
      # availabilityZone: 'IBM ACE Custom Zone' # Broker name will be used by default (optional)
      #######################################################################################################
      # Following parameters are used to access ACE server's pub/sub service, to retrieve performance statistics data
      # Optional parameters are commented out by default. Remove the \u2018#\u2019 when needed.
      # Note: if MQTT is used, then keep mqport only and comment out other MQ parameters.
      #######################################################################################################
      mqHost: 'mqserver.com'        # MQ host/IP address to access channel, which can be reached only by using the external IP address (optional)
      mqport: '1414'                # Set the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM1'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security is enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security is enabled (optional for IBM MQ)
      #mqKeystore: '<MQ_KEYSTORE_PATH>'             # Keystore path for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqKeystorePassword: '<MQ_KEYSTORE_PASSWORD>' # Keystore password for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)
      #mqCipherSuite: '<MQ_CIPHER_SUITE>'           # Cipher suite for TLS connection (optional for IBM MQ. Required only when TLS is enabled for remote monitoring)

Configuring for cloud-native ACE

Configuration for ACE sensor in the cloud-native environment is auto-discovered.

Configuration data

• Default Queue Manager Name
• Product Name
• Version
• Build Level
• Platform Name
• Platform Architecture
• Platform Version
• Application Name

Performance metrics

Known Limitations

• If HTTPS is enabled for REST API interface, you need to specify keystore and keystorePassword parameters in <agent_install_dir>/etc/instana/configuration.yaml. For keystore type, only JKS or P12 is supported.

Health Signatures

Each sensor has a curated knowledgebase of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents, which depend on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any entity.

For more information about built-in events for IBM ACE sensor, see Built-in events reference.

Troubleshooting

• Test the IBM MQ channel connectivity
  If you want to test the IBM MQ channel connectivity, you must set up the IBM MQ Explorer first. Then, you can test the channel connectivity in the IBM MQ Explorer with the credential that is provided in the configuration.yaml file.

• Resolve warning message due to automatic discovery failure
  If the automatic discovery fails, you might get a warning message on your dashboard UI. The explanation and solution for these warning messages are as follows:

  • Auto-discovery: The OS Windows 10 is not supported for automatic discovery.
    This message is displayed if you have OS Windows 10 running on the host. OS Windows 10 is not supported for automatic discovery. To resolve this issue, you must specify integration node information in the <agent_install_dir>/etc/instana/configuration.yaml file to continue monitoring with rest API.
  • Auto-discovery: The OS Windows 10 or the ACE/IIB v10 is not supported for automatic discovery currently.
    This message is displayed when the host either has OS Windows 10 or IBM Integration Bus v10 running. Both of them are not supported. To resolve this issue, follow the same solution as outlined in the previous message.
  • Auto-discovery: Unix Domain Socket is unable to access as the mqsi work path is null.
    This message is displayed when IBM ACE sensor fails to get the WebSphere Message Broker work path or when the WebSphere Message Broker path is null. To resolve this issue, follow the same solution as outlined in the first message.
  • Auto-discovery: MQTT is disabled. You can enable MQTT or specify MQ information in the configuration.yaml file.
    This message is displayed when the MQTT process is disabled or stopped unexpectedly. To resolve this issue, enable MQTT and start the process. If you don't want to enable it, refer to the same solution as outlined in the first message.

• Fix the error code 2058 displayed on connecting to queue manager
  If you cannot get metrics from the Instana UI but instead get an error code 2058 from the agent log, ensure that all IBM MQ related information is correct in the configuration file. Run the ps -ef | grep runmqlsr | grep '<QUEUEMGR NAME>' | grep '<LISTENER_PORT>' command to list the detailed information. If the listener has a specified IP with -i, you must remove the limitation or set up a new listener for ACE.
  See the following example output with correct setting:

  root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1414
  mqm       9956  9872  0 Aug30 ?        00:00:05 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1414
  

  See the following example output with wrong setting:

  root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1417
  mqm       9957 9872  0 Aug30 ?        00:00:06 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1417 -i 10.17.59.113
  

• Obtain the REST API port of the current integration node
  If you want to get the REST API port that the current integration node is using, run the following command:

  # If HTTP protocal is used, get the port with this command
  mqsireportproperties integrationNode -b webadmin -o HTTPConnector -a | grep port
  # If HTTPS protocal is used, get the port with this command
  mqsireportproperties integrationNode  -b webadmin -o HTTPSConnector -a | grep port
  

• Collect the payload in JSON format
  If you want to get the payload in JSON format for further investigation, you can use the following command:

  # CMD for IIBv10
  curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv1?depth=6" >> payload.json
  # CMD for ACEv11
  curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv2?depth=6" >> payload.json
  

• Remove the warning message Monitoring issue: jvm_attach_generic from the integration server dashboard
  This message is displayed when the Instana agent can't correctly attach the DataFlowEngine process, which is a JVM wrapper.

  • If you are using IBM Integration Bus v10, you can ignore this process in <agent_install_dir>/etc/instana/configuration.yaml as follows:

     com.instana.ignore:
       processes:
         - 'DataFlowEngine'
    

  For more information, see Ignore processes.
  Note: The potential impacts of ignoring this process are as follows:
  * The host process of integration server is set to bipservice by default. * The CPU load or usage is improved by ignoring these processes especially for a large-scale environment, which might have many integration servers that are running with the DataFlowEngine process.

  • If you are using IBM App Connect Enterprise v11 or v12, you can choose to ignore this process with the steps as described or fix the warning by completing the following steps:

    1. Set up the JAVA_HOME environment variable, and point it to the ACE bundled JRE by running the following command:

    export JAVA_HOME= <PATH to your ACE Bundled JRE>
    

    See the following example:

    export JAVA_HOME=/opt/ace-11.0.0.11/common/jdk/jre
    

    2. Stop and start the ACE broker (mqsistop/mqsistart).

• Restore missing flow nodes table and the related metrics for self-hosted environment
  A design change was introduced for flow node metrics since release-217. The flow nodes dashboard page is removed, and all flow nodes metrics are displayed with expandable tables. For self-hosted environments, you need to choose one of following solutions to see flow node metrics again:

  • Upgrade the self-hosted version to release-217. With this solution, you can get all the new features of Instana.
  • If you don't want to upgrade the backend to use the features of the old agent, you can pin the agent to version 1.1.620. For more information, see version pinning.
    Note: If you pin the agent to version 1.1.620, all sensors will be on old versions.

• Resolve the error message Create WMQ connection for ACE sensor failed on the Windows Instana agent
  If you don't get the metrics of the Windows agent from the Instana UI, but instead get an error message Create WMQ connection for ACE sensor failed on the agent UI, ensure that the IBM MQ security feature is disabled. If you still get the error message after IBM MQ security is disabled, check whether the IBM MQ user is added to the Windows user group mqm.

Platforms

The Instana ACE tracing user exit supports:

• IBM App Connect Enterprise v11.0.0.8 or later.
• Traditional ACE in Linux x86_64, Linux ppc64le, Linux s390x, AIX 7.2, AIX 7.3, Windows, and container ACE in IBM Cloud Pak for Integration (amd64 only).

Notes:

• On AIX 7.2, you must install XL C/C++ Runtime for AIX 16.1.0 Fix Pack 7 or later versions, including the file set libc++.rte. You can use the following command to check whether the file set libc++.rte is installed:

  lslpp -l |grep libc++.rte
  

• On Windows, you must install MSVC Runtime v2019.

Prerequisites

Enabling tracing for ACE version earlier than v12.0.7 in traditional ACE environment

Enabling tracing when ACE version equal or greater than v12.0.7 in traditional ACE environment

From App Connect Enterprise 12.0.7.0, the native tracing support based on OpenTelemetry is added, and the tracing calls can be accepted by Instana agent's OpenTelemetry ingestion. You don't need to install additional Instana ACE tracing user exit for ACE. You must use ACE native tracing if the ACE version is greater than or equal to 12.0.7.0. Do not enable Instana ACE tracing user exit and ACE native tracing simultaneously because the tracing calls mix with the upstream or downstream tracing calls.

Enabling tracing in CP4I environment

You can use Instana AutoTrace WebHook to enable Instana ACE Tracing in IBM Cloud Pak for Integration (amd64 only). Currently, Instana AutoTrace WebHook supports only App Connect Enterprise 12.0.7.0 or earlier versions. You don't need to install the Instana ACE tracing user exits manually in the Cloud Pak for Integration environment. For more information about how to enable tracing for container ACE that runs in Cloud Pak for Integration, see Instana AutoTrace WebHook - IBM MQ and ACE.

For more information, see Instana AutoTrace Webhook

Troubleshooting Instana ACE tracing

• Check whether Instana ACE tracing is enabled
  To verify whether the Instana ACE tracing is enabled or not, follow the steps:

  1. Check the integration server's stdout file, such as /var/mqsi/components/BK1/servers/EG1/stdout. If the Instana ACE tracing is enabled successfully, you can see the following message:

     Load Tracing Agent in shared mode, 0x7f322822bbbe, (nil)
     /var/mqsi/shared-classes/tracelibrary.so
     Found agent library!
     

  2. Check whether the /tmp/trace directory is generated or not. If the Instana ACE tracing is enabled successfully, the user exit creates the log file like aceExit.13312.log in the /tmp/trace directory. The number 13312 is the DataFlowEngine or Integration Server process ID. See the following example:

     [root@hastened1 trace]# ps -ef | grep 13312
     root     13312 13260 13 18:37 ?        00:00:29 DataFlowEngine v12_BK1 00000000-0000-0000-0000-000000000000 BK1-EG1
     

  3. If Instana ACE tracing fails to be enabled, collect the system log /var/log/message, the integration node's stdout and stderr file, and the integration server's stdout and stderr file. Send all files to Instana support team.

• Missing ACE tracing calls in the Instana UI If you can't see the ACE tracing calls in the Instana UI or tracing call doesn't show as expected, follow the steps:

  1. Enable debugging for Instana ACE tracing.
     • For traditional ACE environment, set the environment variable LOG_LEVEL="debug" in the command line, such as export LOG_LEVEL=debug. Then, in the same command line, run the mqsistop and mqsistart commands to restart ACE instance.
     • For container ACE in IBM Cloud Pak for Integration, set the environment variables LOG_LEVEL as debug for ACE instance. For ACE v11.x, see Managing integration server environment variables in Red Hat OpenShift. For ACE v12.x, see App Connect Integration Server reference - Custom resource values - spec.env.
  2. Restart the Integration Server, and collect all files in the /tmp/trace directory. Collect the tracing call's screen captures on the Instana UI. Send the debug files and screen captures to the Instana Support team.

Supported versions

• Linux and AIX are supported. Windows is not supported.
• IBM App Connect Enterprise v11 or later is supported. IBM Integration Bus v10 is not supported.

Prerequisites

Ensure that the following requirements are met for the automatic discovery feature in ACE:

• ACE sensor is enabled without configuring integration node or server in the <agent_install_dir>/etc/instana/configuration.yaml file.
• Curl is installed on the host. To check the curl installation, run the which curl command.
• Permission is available to access WebSphere Message Broker work path (for example, /var/mqsi/) to start the agent.
• MQTT process is running properly for related integration node. To check, run the ps -ef | grep -E 'bipMQTT.*<INTEGRATION_NODE>’ command.

Note: With automatic discovery, you can see all running integration servers that are listed on the host dashboard of the Instana UI even when all the prerequisites are not met.