Troubleshooting

If you want to test the IBM MQ channel connectivity, you must set up the IBM MQ Explorer first. Then, you can test the channel connectivity in the IBM MQ Explorer with the credential that is provided in the configuration.yaml file.

If the automatic discovery fails, you might get a warning message on your dashboard. The explanation and solution for these warning messages are as follows:

• Auto-discovery: The OS Windows 10 is not supported for automatic discovery.

  This message is displayed if you have OS Windows 10 running on the host. OS Windows 10 is not supported for automatic discovery. To resolve this issue, you must specify integration node information in the <agent_install_dir>/etc/instana/configuration.yaml file to continue monitoring with rest API.

• Auto-discovery: The OS Windows 10 or the ACE/IIB v10 is not supported for automatic discovery currently.

  This message is displayed when the host either has OS Windows 10 or IBM Integration Bus (IIB) 10 running. Both of them are not supported. To resolve this issue, follow the same solution as outlined in the previous message.

• Auto-discovery: Unix Domain Socket is unable to access as the mqsi work path is null.

  This message is displayed when IBM ACE sensor fails to get the WebSphere Message Broker work path or when the WebSphere Message Broker path is null. To resolve this issue, follow the same solution as outlined in the first message.

• Auto-discovery: MQTT is disabled. You can enable MQTT or specify MQ information in the configuration.yaml file.

  This message is displayed when the MQTT process is disabled or stopped unexpectedly. To resolve this issue, enable MQTT and start the process. If you don't want to enable it, refer to the same solution as outlined in the first message.

If you cannot get metrics from the Instana UI but instead get an error code 2058 from the agent log, ensure that all IBM MQ related information is correct in the configuration file. Run the ps -ef | grep runmqlsr | grep '<QUEUEMGR NAME>' | grep '<LISTENER_PORT>' command to list the detailed information. If the listener has a specified IP with -i, you must remove the limitation or set up a new listener for ACE.

See the following sample output with correct setting:

root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1414
mqm       9956  9872  0 Aug30 ?        00:00:05 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1414

See the following sample output with wrong setting:

root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1417
mqm       9957 9872  0 Aug30 ?        00:00:06 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1417 -i 10.17.59.113

If you want to get the REST API port that the current integration node is using, run the following command:

• If HTTP protocol is used, get the port with the following command:

  mqsireportproperties integrationNode -b webadmin -o HTTPConnector -a | grep port
  

• If HTTPS protocol is used, get the port with the following command:

  mqsireportproperties integrationNode  -b webadmin -o HTTPSConnector -a | grep port
  

If you want to get the payload in the JSON format for further investigation, you can use the following commands:

• IIB 10:

  curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv1?depth=6" >> payload.json
  

• ACE 11

  curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv2?depth=6" >> payload.json
  

This message is displayed when the Instana agent can't correctly attach the DataFlowEngine process, which is a JVM wrapper.

• If you are using IIB 10, you can remove this warning with the following steps:

  1. Ignore this process in the <agent_install_dir>/etc/instana/configuration.yaml file as follows:

      com.instana.ignore:
        processes:
          - 'DataFlowEngine'
     

     For more information, see Ignore processes.

  2. Set the forceRemote field to true in the <agent_install_dir>/etc/instana/configuration.yaml file for the IBM ACE sensor.

• If you are using IBM ACE 11 or 12, you can choose to ignore this process with the preceding steps or fix the warning by completing the following steps:

  1. Set up the JAVA_HOME environment variable, and point it to the ACE bundled JRE by running the following command:

     export JAVA_HOME= <PATH to your ACE Bundled JRE>
     

     See the following example:

     export JAVA_HOME=/opt/ace-11.0.0.11/common/jdk/jre
     

  2. Stop and start the ACE broker (mqsistop/mqsistart).

A design change was introduced for flow node metrics since release-217. The flow nodes dashboard page is removed, and all flow nodes metrics are displayed with expandable tables. For self-hosted environments, you need to choose one of the following solutions to see flow node metrics again:

• Upgrade the self-hosted version to release-217. With this solution, you can get all the new features of Instana.
• If you don't want to upgrade the backend to use the features of the old agent, you can pin the agent to version 1.1.620. For more information, see version pinning.

If you don't get the metrics of the Windows agent in the Instana UI, but instead get an error message Create WMQ connection for ACE sensor failed on the agent UI, ensure that the IBM MQ security feature is disabled. If you still get the error message after IBM MQ security is disabled, check whether the IBM MQ user is added to the Windows user group mqm.

To verify whether the Instana ACE Tracing is enabled or not, follow the steps:

1. Check the integration server's stdout file, such as /var/mqsi/components/BK1/servers/EG1/stdout. If the Instana ACE Tracing is enabled successfully, you can see the following message:

   Load Tracing Agent in shared mode, 0x7f322822bbbe, (nil)
   /var/mqsi/shared-classes/tracelibrary.so
   Found agent library!
   

2. Check whether the /tmp/trace directory is generated or not. If the Instana ACE Tracing is enabled successfully, the user exit creates the log file, such as aceExit.13312.log in the /tmp/trace directory. The number 13312 is the DataFlowEngine or Integration Server process ID. See the following example:

   [root@hastened1 trace]# ps -ef | grep 13312
   root     13312 13260 13 18:37 ?        00:00:29 DataFlowEngine v12_BK1 00000000-0000-0000-0000-000000000000 BK1-EG1
   

3. If Instana ACE Tracing fails to be enabled, collect the system log /var/log/message, the integration node's stdout and stderr file, and the integration server's stdout and stderr file. Send all files to the Instana support team.

If you can't see the ACE Tracing calls in the Instana UI or the tracing call doesn't show as expected, complete the following steps:

1. Enable debugging for Instana ACE Tracing.

   • For the traditional ACE environment, set the environment variable LOG_LEVEL="debug" in the command line, such as export LOG_LEVEL=debug .

     Then, in the same command line, run the mqsistop and mqsistart commands to restart the IBM ACE instance.

   • For container ACE in IBM Cloud Pak for Integration, set the environment variables LOG_LEVEL as debug for the IBM ACE instance.

     • For ACE 11.x, see Managing integration server environment variables in Red Hat OpenShift.
     • For ACE 12.x, see App Connect Integration Server reference - Custom resource values - spec.env.

2. Restart the integration server, and collect all files in the /tmp/trace directory and files, such as MqsiTransportUserExitBase.debug.*.log, in the /tmp/ directory.

3. Collect the tracing call's screen captures in the Instana UI.

4. Send the debug files and screen captures to the Instana Support team.