IBM MQ Tracing on IBM i (public preview)

You can trace IBM MQ that runs on IBM i with IBM MQ Tracing. IBM MQ Tracing uses an IBM MQ user exit to trace IBM MQ messages. IBM i does not support local monitoring of IBM MQ, so tracing data is reported to a remote host agent.

If IBM MQ messages set trace context headers (trace correlation information), then tracing data that is collected by the IBM MQ Tracing user exit and other tracers can be correlated. IBM MQ Tracing supports the Instana trace context headers (X_INSTANA_T, X_INSTANA_S, and X_INSTANA_L) and the W3C trace context headers (traceparent and tracestate) for correlating the data. When the IBM MQ Tracing user exit requires propagating trace correlation information, it propagates Instana tracing headers and W3C trace context headers in IBM MQ messages.

Enabling IBM MQ Tracing

To use IBM MQ Tracing for tracing IBM MQ messages, enable IBM MQ Tracing in queue managers.

Prerequisites for enabling IBM MQ Tracing

Ensure that the following requirements are met:

  • Traditional IBM MQ runs on IBM i 7.4 or later.

  • The Open Source Package Management environment is installed on the system. Use it to install the following packages:

    • libopenssl1_1
    • libz1
    • gcc10
    • gcc10-cplusplus

  • IBM MQ server and queue manager instances is installed and configured before you install and configure IBM MQ Tracing.

Download the IBM MQ Tracing user exit

Before you enable tracing, download the IBM MQ Tracing user exit package. To download the IBM MQ Tracing user exit package, complete the following steps:

  1. Download the IBM MQ Tracing user exit .tgz file from the artifactory. To download the file, use an underscore (_) as the username and a valid agent key as the password.
  2. Extract the downloaded .tgz file to a temporary location.
  3. Locate the MQTRACING save file (*SAVF).
  4. Create a library named MQTRACING with a target *SAVF by running the following commands:
    CRTLIB MQTRACING

    CRTSAVF MQTRACING/MQTRACING
  5. Move the downloaded MQTRACING file to the newly created *SAVF in the MQTRACING library. The MQTRACING file contains the following files:
    • WRAPPER.SRVPGM and WRAPPER_R.SRVPGM: IBM MQ user exits that intercept the IBM MQ API calls and call the Portable Application Solutions Environment (PASE) user exit binary.
    • IFSMQTRACE.FILE: A *SAVF that contains the following files:
      • mqtracingexit_r: The IBM MQ user exit binary that is called by WRAPPER and initiates calls to the wrapped OpenTelemetry C++ client library to create spans.
      • tracelibrary.so: The wrapped OpenTelemetry C++ client binary file that provides functions to manage the lifecycle of spans and sends spans to the Instana host agent.
      • mqtracingexit.conf: The mqtracingexit.conf file is the configuration file for all the queue manager instances on the IBM MQ server.
  6. Restore the objects from the MQTRACING file:
    RSTOBJ OBJ(*ALL) SAVLIB(MQTRACING) DEV(*SAVF) SAVF(MQTRACING/MQTRACING)
  7. If the exits64 folder doesn't exist in the mqm directory, create it by running the following command:
    MKDIR DIR('/QIBM/UserData/mqm/exits64')
  8. Restore the objects from the IFSMQTRACE.FILE file:
    RST DEV('/QSYS.LIB/MQTRACING.LIB/IFSMQTRACE.FILE') 
    OBJ(('/QIBM/UserData/mqm/exits64/mqtracingexit_r') 
    ('/QIBM/UserData/mqm/exits64/tracelibrary.so') 
    ('/QIBM/UserData/mqm/exits64/mqtracingexit.conf'))

Enable the IBM MQ Tracing user exit

To enable a user exit, edit the /QIBM/UserData/mqm/mqs.ini or /QIBM/UserData/mqm/qmgrs/<QMNAME>/qm.ini file. The mqs.ini file contains information that is relevant to all the queue manager instances on a host. The qm.ini file contains information that is relevant to a specific queue manager. Depending on whether you want to enable tracing for all the queue manager instances or a specific queue manager, edit the files by following these steps.

Enabling IBM MQ Tracing for all the queue manager instances

To enable tracing for all the queue manager instances on a host, complete the following steps:

  1. Back up the mqs.ini file.

  2. Edit the mqs.ini file.

  3. Enable user exit for all queue manager instances on this node by adding the following content.

    ApiExitCommon:
    Sequence=100
    Function=EntryPoint
    Module=MQTRACING/WRAPPER
    Name=TracingApiExit

  4. Save the changes to the mqs.ini file.

  5. Restart the queue manager instances to apply the changes.

Enabling IBM MQ Tracing for a queue manager

To enable tracing for a specific queue manager, complete the following steps:

  1. Back up the qm.ini file.

  2. Edit the qm.ini file.

  3. Add the following configurations:

    ApiExitLocal:
    Sequence=100
    Function=EntryPoint
    Module=MQTRACING/WRAPPER
    Name=TracingApiExit

    Where:

    • Sequence
      Identifies the sequence of this exit in relation to other exits. An exit with a low sequence number is called before an exit with a higher sequence number.
    • Function
      Identifies the name of the function entry point into the module that contains the user exits code.
    • Module
      Contains the user exits code. If this field contains the full path name of the module, the path is used directly. If this field contains only the module name, the module is located by using the ExitsDefaultPath attribute in the ExitPath settings in the qm.ini file.
    • Name
      Contains the descriptive name of the user exits.

  4. Save changes to the qm.ini file.

  5. Restart the queue manager instances to apply the changes.

For more information about enabling a user exit for IBM MQ, see the IBM MQ guide.

Configuring IBM MQ Tracing

To configure IBM MQ Tracing, complete the following steps:

  1. Go to the /QIBM/UserData/mqm/exits64 directory.

  2. Edit the mqtracingexit.conf file for all the queue manager instances as shown in the following example:

    # configuration for IBM MQ Tracing user exit
LOG_LEVEL="info" #Log level: info, warn, error, debug
MONITOR_LEVEL="normal" #MQ tracing level: normal, off, debug
INSTANA_AGENT_HOST = "<AGENT-HOST>" # instana agent host
INSTANA_AGENT_PORT = 42699 # instana agent port, 42699 will be used by default
INSTANA_AGENT_PROTO = "http" # connection type between mq tracing exit and instana agent, http and https are supported, http will be used by default
HOST_ALIAS="<YOUR-HOST-NAME>" # host name to match with instana agent configuration.yaml

    Where:

    • LOG_LEVEL
      Specifies the log level. Log level can be info, warn, error, or debug. The log files are in the /QIBM/UserData/mqm/trace/<QMGR> directory.
    • MONITOR_LEVEL
      Specifies the default tracing level of the queue managers, which can be set to off, normal, or debug. If MONITOR_LEVEL is set to off, no tracing data is collected. If it is set to normal, the trace correlation information is added to the messages with RFH2 headers or named properties, but is not added to the text messages. If it is set to debug, the trace correlation information is added to all messages that are traced.
    • INSTANA_AGENT_HOST
      Specifies the agent host where the Instana format span data is sent to. IBM i does not support local monitoring of IBM MQ. When you specify a remote agent host, you also need to add a line http.listen=* in the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg file for the remote host agent first because the host agent is not reachable from other hosts by default.
    • INSTANA_AGENT_PORT
      Specifies the port where the Instana agent is listening. The default port is 42699. You need to change the port only when the agent is listening on another port.
    • INSTANA_AGENT_PROTO
      Specifies the type of connection between the IBM MQ Tracing exit and the host agent. IBM MQ sensor supports both HTTP and HTTPS protocols. However, HTTPS is used by default. To use HTTPS, you must first secure the agent endpoint. For more information about how to secure the agent endpoint, see Setting up TLS encryption for agent endpoint.
    • HOST_ALIAS
      Specifies a host alias for the span data that is collected by IBM MQ Tracing user exit. You can link calls to IBM MQ to the infrastructure entity if the IBM MQ sensor also monitors the queue manager. The FQDN of the IBM MQ host is used by default. The host alias value needs to match with the IBM MQ host that is specified in the host agent configuration.yaml file. If the FQDN of the IBM MQ host is not used in the IBM MQ sensor configuration and the host agent is not on the local IBM MQ host, specify a host alias.

  3. Save the file.

  4. Restart queue manager instances.

Upgrading IBM MQ Tracing

When a new version of IBM MQ Tracing is released, you can upgrade IBM MQ Tracing by completing the following steps:

  1. Stop the queue manager instances.
  2. Back up the /QIBM/UserData/mqm/exits64/mqtracingexit.conf file on your IBM MQ server.
  3. Complete the installation instructions, skipping steps 4 and 7.
  4. Replace the /QIBM/UserData/mqm/exits64/mqtracingexit.conf file with the backup version.
  5. Restart the queue managers.

Disabling IBM MQ Tracing

To disable IBM MQ Tracing, change its configuration or disable IBM MQ Tracing user exit.

To disable the collection of tracing data in IBM MQ, set MONITOR_LEVEL to off in the mqtracingexit.conf file.

To disable IBM MQ Tracing user exit, complete the following steps on IBM MQ hosts:

  1. Remove the ApiExitCommon and ApiExitLocal settings that you configured in the /QIBM/UserData/mqm/mqs.ini or /QIBM/UserData/mqm/qmgrs/<QMNAME>/qm.ini file on the IBM MQ hosts where IBM MQ Tracing is enabled.
  2. Restart the queue manager instances.

Troubleshooting

Message processing error caused by extra MQRFH2 headers

To propagate trace correlation information, the IBM MQ Tracing user exit adds extra MQRFH2 headers (X_INSTANA_T, X_INSTANA_S, X_INSTANA_L, traceparent, and tracestate) to the traced messages when the messages already contain MQRFH2 headers or when the monitor level of the message destination is set to debug. But these extra headers cause message processing errors and message rejection in some IBM MQ consumer applications. If the extra MQRFH2 headers cause errors in an IBM MQ client application, do one of the following actions:

  • Modify the IBM MQ client application to ignore the extra MQRFH2 headers that Instana added.
  • Change the value of the PROPCTL property to NONE for the queue that is connected by the IBM MQ client application.

Trace correlation information is not added to the IBM MQ messages

By default, the trace correlation information is automatically added to the IBM MQ messages that contain MQRFH2 headers or message properties. To avoid breaking your IBM MQ client applications, the IBM MQ Tracing user exit doesn't add trace correlation information to plain text messages. If you still want to append the trace correlation information to the messages, you can set MONITOR_LEVEL="debug" in the mqtracingexit.conf file. However, you need to ensure that your IBM MQ client application can parse messages with extra MQRFH2 headers.