IBM MQ Tracing

Enabling tracing for on-premises IBM MQ

lslpp -l |grep libc++.rte

Enabling tracing for IBM MQ on a Red Hat OpenShift cluster

IBM MQ Operator 2.2.0 and later versions natively support Instana IBM MQ Tracing. To get native support, use IBM MQ queue manager container image 9.3.1.0-r2 or later versions. To enable Instana IBM MQ Tracing for queue managers that are deployed by using IBM MQ Operator 2.2.0 or later versions on an IBM Cloud Pak for Integration cluster or a Red Hat OpenShift Container Platform cluster, see. IBM MQ Tracing is supported on Linux on x86_64, ppc64le, and s390x Kubernetes nodes.

You also can use Instana AutoTrace WebHook to enable Instana IBM MQ tracing for IBM MQ on an IBM Cloud Pak for Integration cluster or a Red Hat OpenShift cluster. All versions of IBM MQ queue manager container images are supported. However, Instana AutoTrace WebHook supports only linux/amd64 Kubernetes nodes. For more information about how to enable tracing for IBM MQ container that runs on a Red Hat OpenShift cluster (including IBM Cloud Pak for Integration), see Instana AutoTrace WebHook.

Configuring IBM MQ Tracing for on-premises IBM MQ

To configurie IBM MQ Tracing for on-premises IBM MQ, complete the following steps.

1. Go to the exits64 directory:

   • For Linux and AIX, go to the /var/mqm/exits64 directory.
   • For Windows, go to the C:\ProgramData\IBM\MQ\exits64 directory.

2. Edit the mqtracingexit.conf file.

   # configuration for IBM MQ exit
   LOG_LEVEL="info" #Log level: info, warn, error, debug
   SPAN_FORMAT="instana"
   MONITOR_LEVEL="normal" #MQ tracing level: normal, quiet, off, debug
   IBMMQ_PUBSUB_SUPPORT = "off" #The switch for the publish/subscribe support: on, off
   IBMMQ_RFH2_HEADERS_TO_EXPOSE = "" #A list of message headers that must be included in the `PUT` span payload, and each header is separated by commas.
   INSTANA_AGENT_HOST = "localhost" #(optional)
   INSTANA_AGENT_PORT = 42699 #(optional)
   INSTANA_AGENT_PROTO = "http" #(optional)
   HOST_ALIAS="<YOUR-HOST-NAME>" #(optional)
   

   where:

   • LOG_LEVEL specifies the log level. Log level can be info, warn, error, or debug. The log files are in the /tmp/ibmmqtrace directory on Linux and AIX and in the C:\ProgramData\IBM\MQ\trace directory on Windows.
   • The SPAN_FORMAT variable specifies the trace system to which the span data is sent. Set this variable to instana. IBM MQ Tracing user exit sends span data to the host agent endpoint http://localhost:42699, by default. To send the span data to a remote host agent by using the HTTPS protocol, update the configuration items INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO. The SPAN_FORMAT setting must be the same for the IBM MQ hosts.
   • MONITOR_LEVEL specifies the tracing level of IBM MQ, which can be one of off, quiet, normal, and debug. If MONITOR_LEVEL is set to off, no tracing data is collected. If it is set to quiet, the trace correlation information is not added to all the messages, but the messages can still be traced in the IBM MQ server. If it is set to normal, the trace correlation information is added to the messages with RFH2 headers or named properties, but is not added to text messages. If it is set to debug, the trace correlation information is added to all traced messages. If you encounter the MQRC_HEADER_ERROR (MQRC 2142) error in your message consumer application, you can resolve it by changing the MONITOR_LEVEL property to normal or quiet, or by changing the PROPCTL property of the message destination queues to NONE. Alternatively, to avoid the message header error, you can modify your application to ignore the additional RFH2 header with trace correlation information added by Instana.
   • IBMMQ_PUBSUB_SUPPORT controls whether the support for tracing publish and subscribe messages are enabled. Set it to on to enable the support, and set it to off to disable the support.
   • IBMMQ_RFH2_HEADERS_TO_EXPOSE specifies a list of message headers that must be included in the span payload, and each header is separated by commas. For example, IBMMQ_RFH2_HEADERS_TO_EXPOSE="rfh2_folder1.header1, rfh2_folder2.header2, header3". Any message header with a size larger than 256 bytes are truncated to avoid creating a span size that is too large and affects the overall performance. Additionally, the total length of all the collected message headers in a span must not exceed 3 KB. Therefore, only the most important headers must be configured for the collection.
   • INSTANA_AGENT_HOST specifies the agent host where the Instana format span data is sent to, and localhost is used by default. If you specify a remote agent host, you also need to add a line http.listen=* in <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
   • INSTANA_AGENT_PORT specifies the port where the Instana agent is listening, and 42699 is used by default. You need to change the port only when the agent is listening on another port.
   • INSTANA_AGENT_PROTO specifies the connection type between IBM MQ Tracing exit and the host agent. HTTP and HTTPS protocols are supported and HTTP is used by default. To change the protocol to HTTPS, see Setup TLS Encryption for Agent Endpoint to secure the agent endpoint first.
   • HOST_ALIAS specifies a host alias for the span data that is collected by IBM MQ Tracing user exit. You can link calls to IBM MQ to the infrastructure entity if the IBM MQ sensor also monitors the queue manager. The FQDN of the MQ host is used by default. The host alias value needs to match with the IBM MQ host that is specified in the host agent configuration yaml. If the FQDN of IBM MQ host is not used in the IBM MQ sensor configuration and the host agent is not on the local IBM MQ host, specify a host alias. The IBM MQ sensor can discover the FQDN for the local queue manager instances.

3. Save the file.

4. Rstart queue manager instances.

Note: Instana 246 and later versions support setting a custom service name for IBM MQ Tracing user exit. To set a custom service name, set the INSTANA_SERVICE_NAME environment variable for the IBM MQ processes.

Configuring IBM MQ Tracing on a Red Hat OpenShift cluster

To change the configuration of IBM MQ Tracing for a queue manager running on a Red Hat OpenShift cluster, add new environment variables to the queue manager pod. The environment variable name must match the configuration item name listed in the Configuring IBM MQ Tracing for on-premises IBM MQ section. To add an environment variable for a queue manager pod, complete the following steps:

1. Log in to your Red Hat OpenShift cluster web console.

2. From the navigation panel, click Operators > Installed Operators. All installed Operators in the specified project are displayed.

3. Select IBM MQ Operator. The IBM MQ Operator window is displayed.

4. Go to the queue manager tab. The queue manager Details window is displayed.

5. Select the queue manager for which you want to change the tracing configuration.

6. Go to the YAML tab.

7. Add the environment variable in spec > template > pod > containers > env section. For example, use the following snippet to change the log level of IBM MQ Tracing to debug:

   template:
    pod:
      containers:
        - env:
            - name: LOG_LEVEL
              value: debug
   

8. Save the updated queue manager YAML file.

Upgrading IBM MQ Tracing for on-premises IBM MQ on Linux or AIX

1. Stop the queue manager instances.
2. Back up the /var/mqm/exits64/mqtracingexit.conf file on the IBM MQ server.
3. Extract the downloaded IBM MQ Tracing user exit file and replace the corresponding files in the /var/mqm/exits64 directory.
4. Start the queue manager instances.

Upgrading IBM MQ Tracing for on-premises IBM MQ on Windows

1. Stop the queue manager instances.
2. Back up the C:\ProgramData\IBM\MQ\exits64\mqtracingexit.conf file on the IBM MQ server.
3. Extract the downloaded IBM MQ Tracing user exit file, and replace the corresponding files in the C:\ProgramData\IBM\MQ\exits64 directory.
4. Start the queue manager instances.

Upgrading IBM MQ Tracing on a Red Hat OpenShift cluster

To upgrade IBM MQ Tracing on a Red Hat OpenShift cluster, perform one of the following steps:

• If IBM MQ Tracing is enabled by using IBM MQ Operator 2.2.0 or later versions, upgrade IBM MQ queue manager.
• If IBM MQ Tracing is enabled by using Instana AutoTrace WebHook, deploy the Instana AutoTrace WebHook again, and restart queue manager pods.

Disabling IBM MQ Tracing for on-premises IBM MQ

To disable the collection of tracing data in IBM MQ, set MONITOR_LEVEL to off in the mqtracingexit.con file.

To disable IBM MQ Tracing user exit, complete the following steps on IBM MQ hosts:

1. Remove the ApiExitCommon and ApiExitLocal stanzas that you configured in the /var/mqm/mqs.ini or C:\ProgramData\IBM\MQ\mqs.ini file, and remove the /var/mqm/qmgrs/<QMNAME>/qm.ini or C:\ProgramData\IBM\MQ\qmgrs\<QMNAME>\qm.ini file from the IBM MQ hosts where IBM MQ Tracing is enabled.
2. Restart the queue manager instances.

Disabling IBM MQ Tracing on a Red Hat OpenShift cluster

To disable IBM MQ Tracing on a Red Hat OpenShift cluster, perform one of the following steps:

• If IBM MQ Tracing is enabled by using IBM MQ Operator version 2.2.0 or later, set the Enable Instana tracing toggle to false in IBM Cloud Pak for Integration Platform UI or the OpenShift web console. See the IBM MQ guide.
• If IBM MQ Tracing is enabled by using the Instana AutoTrace WebHook, perform one of the following steps:
  • Uninstall the Instana AutoTrace WebHook, and restart the queue mananger pods.
  • Uninstall the Instana AutoTrace WebHook, set autotrace.ibmmq.enabled to false, and then deploy Instana AutoTrace WebHook again. For the changes to take effect, restart the queue manager pods. See Instana AutoTrace WebHook guide.