Troubleshooting

Learn how to resolve issues with monitoring IBM App Connect Enterprise (ACE).

Instana IBM ACE
Instana IBM ACE Tracing
- Check whether Instana ACE Tracing is enabled
- Missing ACE Tracing calls in the Instana UI

Instana IBM ACE

Test the IBM MQ channel connectivity

If you want to test the IBM MQ channel connectivity, you must set up the IBM MQ Explorer first. Then, you can test the channel connectivity in the IBM MQ Explorer with the credential that is provided in the configuration.yaml file.

Resolve warning message due to automatic discovery failure

If the automatic discovery fails, you might get a warning message on your dashboard. The explanation and solution for these warning messages are as follows:

Auto-discovery: The OS Windows 10 is not supported for automatic discovery.

This message is displayed if you have OS Windows 10 running on the host. OS Windows 10 is not supported for automatic discovery. To resolve this issue, you must specify integration node information in the <agent_install_dir>/etc/instana/configuration.yaml file to continue monitoring with rest API.
Auto-discovery: The OS Windows 10 or the ACE/IIB v10 is not supported for automatic discovery currently.

This message is displayed when the host either has OS Windows 10 or IBM Integration Bus (IIB) 10 running. Both of them are not supported. To resolve this issue, follow the same solution as outlined in the previous message.
Auto-discovery: Unix Domain Socket is unable to access as the mqsi work path is null.

This message is displayed when IBM ACE sensor fails to get the WebSphere Message Broker work path or when the WebSphere Message Broker path is null. To resolve this issue, follow the same solution as outlined in the first message.
Auto-discovery: MQTT is disabled. You can enable MQTT or specify MQ information in the configuration.yaml file.

This message is displayed when the MQTT process is disabled or stopped unexpectedly. To resolve this issue, enable MQTT and start the process. If you don't want to enable it, refer to the same solution as outlined in the first message.

Fix the error code 2058 displayed on connecting to the queue manager

If you cannot get metrics from the Instana UI but instead get an error code 2058 from the agent log, ensure that all IBM MQ related information is correct in the configuration file. Run the ps -ef | grep runmqlsr | grep '<QUEUEMGR NAME>' | grep '<LISTENER_PORT>' command to list the detailed information. If the listener has a specified IP with -i, you must remove the limitation or set up a new listener for ACE.

See the following sample output with correct setting:

root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1414
mqm       9956  9872  0 Aug30 ?        00:00:05 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1414

See the following sample output with wrong setting:

root@upstairs1:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1417
mqm       9957 9872  0 Aug30 ?        00:00:06 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1417 -i 10.17.59.113

Obtain the REST API port of the current integration node

If you want to get the REST API port that the current integration node is using, run the following command:

If HTTP protocol is used, get the port with the following command:

mqsireportproperties integrationNode -b webadmin -o HTTPConnector -a | grep port

If HTTPS protocol is used, get the port with the following command:

mqsireportproperties integrationNode  -b webadmin -o HTTPSConnector -a | grep port

Collect the payload in JSON format

If you want to get the payload in the JSON format for further investigation, you can use the following commands:

IIB 10:

curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv1?depth=6" >> payload.json

ACE 11

curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv2?depth=6" >> payload.json

Remove the warning message `Monitoring issue: jvm_attach_generic` from the integration server dashboard

This message is displayed when the Instana agent can't correctly attach the DataFlowEngine process, which is a JVM wrapper.

If you are using IIB 10, you can remove this warning with the following steps:
1. Ignore this process in the <agent_install_dir>/etc/instana/configuration.yaml file as follows:
```
 com.instana.ignore:
   processes:
     - 'DataFlowEngine'
```
  For more information, see Ignore processes.
2. Set the forceRemote field to true in the <agent_install_dir>/etc/instana/configuration.yaml file for the IBM ACE sensor.
If you are using IBM ACE 11 or 12, you can choose to ignore this process with the preceding steps or fix the warning by completing the following steps:
1. Set up the JAVA_HOME environment variable, and point it to the ACE bundled JRE by running the following command:
```
export JAVA_HOME= <PATH to your ACE Bundled JRE>
```
  See the following example:
```
export JAVA_HOME=/opt/ace-11.0.0.11/common/jdk/jre
```
2. Stop and start the ACE broker (mqsistop/mqsistart).

Resolve the error message `Create WMQ connection for ACE sensor failed` on the Windows Instana agent

If you don't get the metrics of the Windows agent in the Instana UI, but instead get an error message Create WMQ connection for ACE sensor failed on the agent UI, ensure that the IBM MQ security feature is disabled. If you still get the error message after IBM MQ security is disabled, check whether the IBM MQ user is added to the Windows user group mqm.

Instana IBM ACE Tracing

Check whether Instana ACE Tracing is enabled

To verify whether the Instana ACE Tracing is enabled or not, follow the steps:

Check the integration server's stdout file, such as /var/mqsi/components/BK1/servers/EG1/stdout. If the Instana ACE Tracing is enabled successfully, you can see the following message:
```
Load Tracing Agent in shared mode, 0x7f322822bbbe, (nil)
/var/mqsi/shared-classes/tracelibrary.so
Found agent library!
```
Check whether the /tmp/trace directory is generated or not. If the Instana ACE Tracing is enabled successfully, the user exit creates the log file, such as aceExit.13312.log in the /tmp/trace directory. The number 13312 is the DataFlowEngine or Integration Server process ID. See the following example:
```
[root@hastened1 trace]# ps -ef | grep 13312
root     13312 13260 13 18:37 ?        00:00:29 DataFlowEngine v12_BK1 00000000-0000-0000-0000-000000000000 BK1-EG1
```
If Instana ACE Tracing fails to be enabled, collect the system log /var/log/message, the integration node's stdout and stderr file, and the integration server's stdout and stderr file. Send all files to the Instana support team.

Missing ACE Tracing calls in the Instana UI

If you can't see the ACE Tracing calls in the Instana UI or the tracing call doesn't show as expected, complete the following steps:

Enable debugging for Instana ACE Tracing.
- For the traditional ACE environment, set the environment variable LOG_LEVEL="debug" in the command line, such as export LOG_LEVEL=debug.
  
  Then, in the same command line, run the mqsistop and mqsistart commands to restart the IBM ACE instance.
- For container ACE in IBM Cloud Pak for Integration, set the environment variables LOG_LEVEL as debug for the IBM ACE instance.
  - For ACE 11.x, see Managing integration server environment variables in Red Hat OpenShift.
  - For ACE 12.x, see App Connect Integration Server reference - Custom resource values - spec.env.
Restart the integration server, and collect all files in the /tmp/trace directory and files, such as MqsiTransportUserExitBase.debug.*.log, in the /tmp/ directory.
Collect the tracing call's screen captures in the Instana UI.
Send the debug files and screen captures to the Instana Support team.