Troubleshooting IBM MQ

For the queue managers that are running, after you make the user privileged, make sure that the security settings work. You must either refresh security by running the following IBM MQ runmqsc command or restart queue manager to make authority that you granted take effective.

REFRESH SECURITY
 

For the client binding mode, the IBM MQ sensor acts as a client application to connect to queue manager. Therefore, the sensor requires the same authority as any other applications to connect to queue manager and inquire data, such as MQ explorer, amqsputc MQ sample application. When queue manager does not get connected with your configured parameters, you can try your connection parameters with MQ explorer to connect to queue manager and check whether you can connect queue manager.

See the following issues for reference:

1. RC 2538 - MQRC_HOST_NOT_AVAILABLE

   Instana agent log message: Listener not started on{host}:{port} ({exceptionCode}) {message}.

   • Listener is not started.

     Solution: Start the listener. To start the listener, run the following IBM MQ runmqsc command:

     START LISTENER($Listener_Name)
      

   • Qmgr@host can’t be found. Check if the queue manager and host names are correct and can be connected.

     Solution: Check if the connection parameters for queue manager and host names are correct and can be connected. Or check if a firewall is blocking the connection.

   • The host and port are configured in the agent configuration file in the Kubernetes cluster.

     In the Kubernetes cluster, when queue manager is restarted, the host IP changes. Therefore, the IBM MQ sensor fails to connect to queue manager.

     Solution: To automatically discover the host and port information, delete the host and port in the agent configuration file for the host agent in the Kubernetes cluster.

2. RC 2540 - Channel is not defined.

   Instana agent log message: Channel {channel} is not defined ({exceptionCode}). {message}.

   Solution: Check if the correct SVRCONN channel is configured in the configuration.yaml agent configuration file.

3. RC 2035 - Authority problem.

   Instana agent log message: Channel {channel} authorization failed for user {username} ({exceptionCode}). {message}.

   The problem is caused by authority problem and might be caused by different reasons according to different MQ configurations. To debug the authority problem, check whether channel security is enabled:

   • If channel security is disabled, but reports no authority with the user, then the MQ Application asserts user has no authority. Check which user is used and provide the correct authority to this user or change to another proper user to connect to queue manager.
   • If channel security is enabled. Check whether the provided user and password have the correct authority to connect to queue manager. If the TLS is enabled, you need to provide the correct keystore, keystorePassword, and the corresponding cipherSuite to connect queue manager.

The following image shows the debug flow chart:

The following problems that are frequently encountered might cause 2035 error:

1. Channel security is disabled, but no authority with “root” is reported.

   Usually when CHLAUTH parameter is disabled and CONNAUTH parameter is not set, then the MQ channel security is disabled. You need to determine which user is used for authorization. Here is the order of precedence for security features.

   If you do not configure Security exit, channel record USERSRC(MAP), CLNTUSER, MCAUSER, then the application asserted user, which is the operating system user in a remote connection is used. In this scenario, “root” is used as the application asserted user because the Instana agent is running as root. If root doesn’t have authority, you obtain no authority with “root” in the IBM MQ error log. For more user authority priority details, see IBM MQ Documents.

   Solution: You can configure the channel record and define CLNTUSER or MCAUSER for the channel. Define MCAUSER as a user with MQ access authority for your server connection channel, and then use this user to connect queue manager.

   For example, alter channel(SVRCONN) chltype(SVRCONN) MCAUSER(‘mqmtest‘)

2. No authority is granted to connect to the SYSTEM server connection channel.

   Some system server connection channels like SYSTEM.AUTO.SVRCONN are blocked by default, here are MQ BLOCKUSERS rules: There are three default rules for CHLAUTH processing:

   • NO ACCESS to all channels by any MQ-admin* users
   • NO ACCESS to all SYSTEM.* channels by all users
   • ALLOW access to SYSTEM.ADMIN.SVRCONN channel (non MQ-admin users)

   The first two rules block access to all channels. The third rule is more specific and takes preference over the other two, that is, CHLAUTH allows access only to SYSTEM.ADMIN.SVRCONN channel. For more information, see Resolving CHLAUTH access issues.

   Solution: Unblock the user for system server connection channel before the user is used, or define your own server connection channel for connection, which can workaround this block user problem.

3. Security is enabled or TLS is enabled

   When channel security is enabled--CHLAUTH(ENABLED), or TLS is enabled, but the corresponding username and password or keystore parameters (keystore, keystorePassword, and cipherSuite) are not provided, then provide the credentials. After you change MQ security-related configurations, you need to run refresh security type(CONNAUTH) from the runmqsc prompt to make it work.

   Solution: Provide the corresponding username and password or keystore parameters in the agent configuration file.

4. Security is enabled for both CHLAUTH and CONNAUTH, but user still has authority problem to connect to queue manager.

   Solution: CHLAUTH and CONNAUTH configurations in queue manager, and then check CHLAUTH and CONNAUTH interactions flow to confirm the priority to see which security record works, and correct the problem. For more information, see Interaction of CHLAUTH and CONNAUTH.

5. MQ connection has no problem, but object security is not enough for the user to get other monitoring data.

   Solution: Check the MQ error log to see which object and which user have an authority problem, and provide the correct authority to the object to solve the problem. For more information, see Configuring IBM MQ authority.

You encounter the following error when the IBM MQ has insufficient authority to obtain the queue reset statistics data. To obtain queue reset statistics data, you need the change authority to access SYSTEM.AUTH.DATA.QUEUE. But SYSTEM.AUTH.DATA.QUEUE is a special queue for which you cannot provide permission to change authority.

AMQ8077W: Entity 'user' has insufficient authority to access object
'SYSTEM.AUTH.DATA.QUEUE'.

EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: chg
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.
 

Solution: If you want to stop such error messages from appearing in the IBM MQ log for SYSTEM.AUTH.DATA.QUEUE, make the user of the Instana agent privileged.

To troubleshoot, complete the following steps:

1. Ensure that tracelibrary.so (on Linux/AIX) or tracelibrary.dll (on Windows) can resolve all dependencies for all users that start IBM MQ services, including the mqm user on Linux or AIX and the MQ_USER on Windows. To verify, use the following commands:

   • Linux: ldd
   • AIX: dump -H or ldd
   • Windows:dumpbin /dependents
2. Confirm that all required files are present in the expected locations:
   • Linux: .so
   • AIX: .a or .so
   • Windows: .dll
3. Verify that the system has the required OpenSSL version installed and that tracelibrary.so or tracelibrary.dll can resolve it at runtime.

To propagate trace correlation information, IBM MQ Tracing user exit adds additional MQRFH2 headers (X_INSTANA_T, X_INSTANA_S, X_INSTANA_L, traceparent, and tracestate) to the messages that are being traced when the messages already contain MQRFH2 headers or when the monitor level of the message destination is set to debug. But the presence of additional header data in messages can cause message processing errors and message rejection in some IBM MQ consumer applications. If the additional MQRFH2 headers cause errors in an IBM MQ client application, do one of the following actions:

• Modify the IBM MQ client application to ignore the additional MQRFH2 headers with trace correlation information that Instana added.
• Disable trace correlation propagation by adding the queues in the messaging chain to the IBM MQ Tracing user exit configuration parameter IBMMQ_DEST_MONITOR_LEVEL_QUIET.
• Change the value of PROPCTL property to NONE for the queue connected by the IBM MQ client application.

By default, the trace correlation information is automatically added to the IBM MQ messages that contain MQRFH2 headers or message properties. To avoid breaking your IBM MQ client applications, IBM MQ Tracing user exit doesn't add trace correlation information to plain text messages. If you still want to append the trace correlation information to the messages, you can add the message destinations to the IBM MQ Tracing user exit configuration parameter IBMMQ_DEST_MONITOR_LEVEL_DEBUG. However, you need to ensure that your IBM MQ client application can parse messages with additional MQRFH2 headers.

You can link a call that is generated by the IBM MQ Tracing user exit to a monitored infrastructure entity of the IBM MQ queue manager, which is collected by the Instana IBM MQ sensor in the Instana UI. The infrastructure link might not be available in IBM MQ high-availability scenarios because the entity ID of a queue manager that is collected by the Instana IBM MQ sensor contains the host of both active and standby IBM MQ nodes. This contrasts with the entity ID generated by the IBM MQ Tracing user exit, which only includes the host of the active IBM MQ node. The mismatch disrupts the link between call details and the queue entity. To resolve this issue, you can take the following actions:

1. Go to the high-availability queue manager dashboard within the Infrastructure view in the Instana UI.
2. Locate the Name attribute in the sidebar of the queue manager dashboard to extract the queue manager host information. The host for the high-availability queue manager is the part of the Name string that comes after the @ symbol. For example, if the queue manager name appears as haqm1@ibmmqnode1.ibm.com-ibmmqnode2.ibm.com-ibmmqnode3.ibm.com, then the queue manager host is ibmmqnode1.ibm.com-ibmmqnode2.ibm.com-ibmmqnode3.ibm.com.
3. Access the IBM MQ server, and open the mqtracingexit.conf configuration file for IBM MQ Tracing user exit.
4. Set the HOST_ALIAS field to the queue manager hostname that is obtained in step 2.
5. Save the configuration file.
6. Restart the high-availability queue manager to apply the changes.