IBM MQ Tracing

Enabling tracing for on-premises IBM MQ

lslpp -l |grep libc++.rte

Enabling tracing for IBM MQ on a Red Hat OpenShift cluster

IBM MQ Operator 2.2.0 and later versions natively support Instana IBM MQ Tracing. To get native support, use IBM MQ queue manager container image 9.3.1.0-r2 or later versions. To enable Instana IBM MQ Tracing for queue managers that are deployed by using IBM MQ Operator 2.2.0 or later versions on an IBM Cloud Pak for Integration cluster or a Red Hat OpenShift Container Platform cluster, see. IBM MQ Tracing is supported on Linux on x86_64, ppc64le, and s390x Kubernetes nodes.

To enable Instana IBM MQ Tracing by using IBM MQ Operator on a Red Hat OpenShift Container Platform cluster, you must use an IBM Cloud Pak for Integration license with the IBM MQ Operator.

You can also use Instana AutoTrace webhook to enable Instana IBM MQ Tracing for the IBM MQ on an IBM Cloud Pak for Integration cluster. Instana AutoTrace webhook supports only IBM MQ queue manager container image versions earlier to 9.3.3.0 and Linux/amd64 Kubernetes nodes. For more information about how to enable tracing for IBM MQ containers that run on an IBM Cloud Pak for Integration cluster, see Instana AutoTrace webhook.

Configuring IBM MQ Tracing for on-premises IBM MQ

To configurie IBM MQ Tracing for on-premises IBM MQ, complete the following steps.

1. Go to the exits64 directory:

   • For Linux and AIX, go to the /var/mqm/exits64 directory.
   • For Windows, go to the C:\ProgramData\IBM\MQ\exits64 directory.

2. Edit the mqtracingexit.conf file for all the queue manager instances or the mqtracingexit_<QMNAME>.conf file for specific queue manager.

   # configuration for IBM MQ Tracing user exit
   LOG_LEVEL="info" #Log level: info, warn, error, debug
   SPAN_FORMAT="instana" #Span Format: instana, otel
   MONITOR_LEVEL="normal" #MQ tracing level: normal, quiet, off, debug
   IBMMQ_DEST_MONITOR_LEVEL_OFF = "" Regex for message destinations with tracing level off, for example: "^TMPQ.*". An example for multiple conditions: "^SYSTEM\\.AUTH\\..*|^TEMP\\.QUEUE\\..*".
   IBMMQ_DEST_MONITOR_LEVEL_QUIET = "" #Regex for message destinations with tracing level quiet
   IBMMQ_DEST_MONITOR_LEVEL_NORMAL = "" #Regex for message destinations with tracing level normal 
   IBMMQ_DEST_MONITOR_LEVEL_DEBUG = "" #Regex for message destinations with tracing level debug
   IBMMQ_PUBSUB_SUPPORT = "off" #The switch for the publish/subscribe support: on, off
   IBMMQ_RFH2_HEADERS_TO_EXPOSE = "" #A list of message headers that must be included in the `PUT` span payload, and each header is separated by commas.
   INSTANA_AGENT_HOST = "localhost" #(optional) Applies only when the SPAN_FORMAT is set to instana
   INSTANA_AGENT_PORT = 42699 #(optional) Applies only when the SPAN_FORMAT is set to instana
   INSTANA_AGENT_PROTO = "http" #(optional) Applies only when the SPAN_FORMAT is set to instana
   HOST_ALIAS="<YOUR-HOST-NAME>" #(optional) Applies only when the SPAN_FORMAT is set to instana
   LOCAL_SOURCE_ADDRESS = "" #(optional) The local source ip address for outgoing traffic
   OTLP_EXPORTER_GRPC_ENDPOINT = "" #(optional) Applies only when the SPAN_FORMAT is set to otel
   OTLP_EXPORTER_SSL_CACERT_PATH = "" #(optional) Applies only when the SPAN_FORMAT is set to otel
   OTLP_EXPORTER_HTTP_URL = "" #(optional) Applies only when the SPAN_FORMAT is set to otel
   

   where:

   • LOG_LEVEL
     This parameter specifies the log level. Log level can be info, warn, error, or debug. On Linux and AIX, the log files are in the /tmp/ibmmqtrace directory, and on Windows, the log files are in the C:\ProgramData\IBM\MQ\trace directory. To change the default directory of the log files, see the steps in the Log files for IBM MQ Tracing section.
   • SPAN_FORMAT
     This parameter specifies the trace system to which the span data is sent. Set this variable to instana. By default, IBM MQ Tracing user exit sends span data to the host agent endpoint http://localhost:42699. To send the span data to a remote host agent by using the HTTPS protocol, update the configuration items INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO. The SPAN_FORMAT setting must be the same for the IBM MQ hosts.
   • MONITOR_LEVEL
     This parameter specifies the default tracing level of the queue managers, which can be set to off, quiet, normal, or debug. If MONITOR_LEVEL is set to off, no tracing data is collected. If it is set to quiet, the trace correlation information is not added to all the messages, but the messages can still be traced in the IBM MQ server. If it is set to normal, the trace correlation information is added to the messages with RFH2 headers or named properties, but is not added to the text messages. If it is set to debug, the trace correlation information is added to all messages that are being traced.
   • IBMMQ_DEST_MONITOR_LEVEL_OFF
     This parameter specifies the message destinations with the tracing level off. The value of this parameter must be a regular expression, such as "^TMPQ.*". It means that the messages that are sent to the destinations that start with TMPQ are not traced. To ensure the integrity of the transaction, you must keep the same tracing level for message destinations in the same messaging chain. If a message destination matches multiple regular expressions, then the priority order of the effective regular expressions is as follows: IBMMQ_DEST_MONITOR_LEVEL_OFF, IBMMQ_DEST_MONITOR_LEVEL_QUIET, IBMMQ_DEST_MONITOR_LEVEL_NORMAL, and IBMMQ_DEST_MONITOR_LEVEL_DEBUG.
   • IBMMQ_DEST_MONITOR_LEVEL_QUIET
     This parameter specifies the message destinations with the tracing level quiet. The value of this parameter must be a regular expression. If a message destination matches this regular expression, then the trace correlation information is not added to all the messages that are sent to the message destination. But you can still trace the messages in the IBM MQ server.
   • IBMMQ_DEST_MONITOR_LEVEL_NORMAL
     This parameter specifies the message destinations with the tracing level normal. The value of this parameter must be a regular expression. If the message destination matches this regular expression, then the trace correlation information is added to the messages with RFH2 headers or named properties. But the trace correlation information is not added to the text messages.
   • IBMMQ_DEST_MONITOR_LEVEL_DEBUG
     This parameter specifies the message destinations with the tracing level debug. The value of this parameter must be a regular expression. If the message destination matches this regular expression, then the trace correlation information is added to all the messages that are sent to the message destination.
   • IBMMQ_PUBSUB_SUPPORT
     This parameter controls whether the support for tracing publish and subscribe messages are enabled. Set it to on to enable and off to disable the support.
   • IBMMQ_RFH2_HEADERS_TO_EXPOSE
     This parameter specifies a list of message headers that must be included in the span payload, and each header is separated by a comma, for example, IBMMQ_RFH2_HEADERS_TO_EXPOSE="rfh2_folder1.header1, rfh2_folder2.header2, header3". Any message header with a size larger than 256 bytes is truncated to avoid creating a span size that is too large and affects the overall performance. Also, the total length of all the collected message headers in a span must not exceed 3 KB. Therefore, only the most important headers must be configured for the collection.
   • INSTANA_AGENT_HOST
     This parameter specifies the agent host where the Instana format span data is sent to, and localhost is used by default. If you specify a remote agent host, you also need to add a line http.listen=* in <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
   • INSTANA_AGENT_PORT
     This parameter specifies the port where the Instana agent is listening, and 42699 is used by default. You need to change the port only when the agent is listening on another port.
   • INSTANA_AGENT_PROTO
     This parameter specifies the connection type between the IBM MQ Tracing exit and the host agent. IBM MQ sensor supports both HTTP and HTTPS protocols. However, HTTP is used by default. To change the protocol to HTTPS, you must first secure the agent endpoint. For more information about how to secure the agent endpoint, see Setting up TLS encryption for agent endpoint.
   • HOST_ALIAS
     This parameter specifies a host alias for the span data that is collected by IBM MQ Tracing user exit. You can link calls to IBM MQ to the infrastructure entity if the IBM MQ sensor also monitors the queue manager. The FQDN of the MQ host is used by default. The host alias value needs to match with the IBM MQ host that is specified in the host agent configuration yaml. If the FQDN of IBM MQ host is not used in the IBM MQ sensor configuration and the host agent is not on the local IBM MQ host, specify a host alias. The IBM MQ sensor can discover the FQDN for the local queue manager instances.
   • LOCAL_SOURCE_ADDRESS This parameter specifies the local source IP address for outgoing traffic from IBM MQ Tracing user exit. You need to configure it only if your IBM MQ server has multiple network interfaces and you want to bind all outgoing traffic from IBM MQ Tracing user exit to a specific network interface.

3. Save the file.

4. Restart queue manager instances.

Instana 246 and later versions support setting a custom service name for IBM MQ Tracing user exit. To set a custom service name, set the INSTANA_SERVICE_NAME environment variable for the IBM MQ processes.

Configuring IBM MQ Tracing on a Red Hat OpenShift cluster

To change the configuration of IBM MQ Tracing for a queue manager running on a Red Hat OpenShift cluster, add new environment variables to the queue manager pod. The environment variable name must match the configuration item name listed in the Configuring IBM MQ Tracing for on-premises IBM MQ section. To add an environment variable for a queue manager pod, complete the following steps:

1. Log in to your Red Hat OpenShift cluster web console.

2. From the navigation panel, click Operators > Installed Operators. All installed Operators in the specified project are displayed.

3. Select IBM MQ Operator. The IBM MQ Operator window is displayed.

4. Go to the queue manager tab. The queue manager Details window is displayed.

5. Select the queue manager for which you want to change the tracing configuration.

6. Go to the YAML tab.

7. Add the environment variable in spec > template > pod > containers > env section. For example, use the following snippet to change the log level of IBM MQ Tracing to debug:

   template:
    pod:
      containers:
        - env:
            - name: LOG_LEVEL
              value: debug
   

8. Save the updated queue manager YAML file.

Upgrading IBM MQ Tracing for on-premises IBM MQ on Linux or AIX

1. Stop the queue manager instances and the Instana agent if the Instana agent is running on the IBM MQ server.
2. Back up the configuration files for the IBM MQ Tracing user exit in the /var/mqm/exits64 directory on the IBM MQ server.
3. Extract the downloaded IBM MQ Tracing user exit file and replace the corresponding files in the /var/mqm/exits64 directory.
4. Start the queue manager instances and the Instana agent if the Instana agent was stopped in step 1.

Upgrading IBM MQ Tracing for on-premises IBM MQ on Windows

1. Stop the queue manager instances, MQ explorer, and the Instana agent if the Instana agent is running on the IBM MQ server.
2. Back up the configuration files for the IBM MQ Tracing user exit in the C:\ProgramData\IBM\MQ\exits64 directory on the IBM MQ server.
3. Extract the downloaded IBM MQ Tracing user exit file, and replace the corresponding files in the C:\ProgramData\IBM\MQ\exits64 directory.
4. Start the queue manager instances, MQ explorer, and the Instana agent if the Instana agent was stopped in step 1.

Upgrading IBM MQ Tracing on a Red Hat OpenShift cluster

To upgrade IBM MQ Tracing on a Red Hat OpenShift cluster, perform one of the following steps:

• If IBM MQ Tracing is enabled by using IBM MQ Operator 2.2.0 or later versions, upgrade IBM MQ queue manager.
• If IBM MQ Tracing is enabled by using Instana AutoTrace webhook, deploy the Instana AutoTrace webhook again, and restart queue manager pods.

Disabling IBM MQ Tracing for on-premises IBM MQ

To disable the collection of tracing data in IBM MQ, set MONITOR_LEVEL to off in the mqtracingexit.con file.

To disable IBM MQ Tracing user exit, complete the following steps on IBM MQ hosts:

1. Remove the ApiExitCommon and ApiExitLocal stanzas that you configured in the /var/mqm/mqs.ini or C:\ProgramData\IBM\MQ\mqs.ini file, and remove the /var/mqm/qmgrs/<QMNAME>/qm.ini or C:\ProgramData\IBM\MQ\qmgrs\<QMNAME>\qm.ini file from the IBM MQ hosts where IBM MQ Tracing is enabled.
2. Restart the queue manager instances, MQ Explorer, and the Instana agent if the Instana agent is running on the IBM MQ server.

Disabling IBM MQ Tracing on a Red Hat OpenShift cluster

To disable IBM MQ Tracing on a Red Hat OpenShift cluster, perform one of the following steps:

• If IBM MQ Tracing is enabled by using IBM MQ Operator version 2.2.0 or later, set the Enable Instana tracing toggle to false in IBM Cloud Pak for Integration Platform UI or the OpenShift web console. See the IBM MQ guide.
• If IBM MQ Tracing is enabled by using the Instana AutoTrace webhook, perform one of the following steps:
  • Uninstall the Instana AutoTrace webhook, and restart the queue manager pods.
  • Uninstall the Instana AutoTrace webhook, set autotrace.ibmmq.enabled to false, and then deploy Instana AutoTrace webhook again. For the changes to take effect, restart the queue manager pods. See Instana AutoTrace webhook guide.

Message processing error caused by extra MQRFH2 headers

To propagate trace correlation information, IBM MQ Tracing user exit adds extra MQRFH2 headers (X_INSTANA_T, X_INSTANA_S, X_INSTANA_L, traceparent, and tracestate) to the messages that are being traced when the messages already contain MQRFH2 headers or when the monitor level of the message destination is set to debug. But the presence of extra header data in messages can cause message processing errors and message rejection in some IBM MQ consumer applications. If the extra MQRFH2 headers cause errors in an IBM MQ client application, do one of the following actions:

• Modify the IBM MQ client application to ignore the extra MQRFH2 headers with trace correlation information that Instana added.
• Disable trace correlation propagation by adding the queues in the messaging chain to the IBM MQ Tracing user exit configuration parameter IBMMQ_DEST_MONITOR_LEVEL_QUIET.
• Change the value of PROPCTL property to NONE for the queue connected by the IBM MQ client application.

Trace correlation information is not added to the IBM MQ messages

By default, the trace correlation information is automatically added to the IBM MQ messages that contain MQRFH2 headers or message properties. To avoid breaking your IBM MQ client applications, IBM MQ Tracing user exit doesn't add trace correlation information to plain text messages. If you still want to append the trace correlation information to the messages, you can add the message destinations to the IBM MQ Tracing user exit configuration parameter IBMMQ_DEST_MONITOR_LEVEL_DEBUG. However, you need to ensure that your IBM MQ client application can parse messages with extra MQRFH2 headers.

The infrastructure link is not displayed in the call details

You can link a call that is generated by the IBM MQ Tracing user exit to a monitored infrastructure entity of the IBM MQ queue manager, which is collected by the Instana IBM MQ sensor in the Instana UI. The infrastructure link might not be available in IBM MQ high-availability scenarios because the entity ID of a queue manager that is collected by the Instana IBM MQ sensor contains the host of both active and standby IBM MQ nodes. This contrasts with the entity ID generated by the IBM MQ Tracing user exit, which only includes the host of the active IBM MQ node. The mismatch disrupts the link between call details and the queue entity. To resolve this issue, you can take the following actions:

1. Go to the high-availability queue manager dashboard within the Infrastructure view in the Instana UI.
2. Locate the Name attribute in the sidebar of the queue manager dashboard to extract the queue manager host information. The host for the high-availability queue manager is the part of the Name string that comes after the @ symbol. For example, if the queue manager name appears as "haqm1@ibmmqnode1.ibm.com-ibmmqnode2.ibm.com-ibmmqnode3.ibm.com", then the queue manager host is "ibmmqnode1.ibm.com-ibmmqnode2.ibm.com-ibmmqnode3.ibm.com".
3. Access the IBM MQ server, and open the mqtracingexit.conf configuration file for IBM MQ Tracing user exit.
4. Set the HOST_ALIAS field to the queue manager hostname that is obtained in step 2.
5. Save the configuration file.
6. Restart the high-availability queue manager to apply the changes.