Message flow statistics and accounting data

Message flow statistics and accounting data can be collected to record performance and operating details of one or more message flows.

You can use message flow statistics and accounting data to capture dynamic information about the runtime behavior of a message flow. For example, it indicates how many messages are processed and how large those messages are, processor usage and elapsed processing times. The statistical data is collected and recorded in a specified location when an event occurs. For example, when a snapshot interval expires or when the integration server that you are collecting information about stops. You can configure the data to be published in JSON or XML formats, or to be written in Comma Separated Values (CSV) format so that you can use it in a spreadsheet.

You can publish statistics and accounting data to one or more of the following destinations:
  • The App Connect Enterprise web user interface (independent integration servers and integration servers that are managed by an integration node)
  • Files
  • User trace
  • IBM MQ
  • MQTT
You can use the statistics that are generated for the following purposes:
  • You can use snapshot data to assess the execution of a message flow to determine why it, or a node within it, is not performing as you expect.
  • You can determine the route that messages are taking through a message flow. For example, you might find that an error path is taken more frequently than you expect. You can use the statistics to understand when the messages are routed to this error path.

    Check the information provided by snapshot data for routing information. If this information is insufficient for your needs, use archive data.

  • You can record the load that applications, business partners, or other users put on the integration server. You can use this information to calculate what to charge users based on their relative use of the integration server. For example, you might levy a nominal charge on every message that is processed by an integration server, or by a specific message flow. You can use archive data to make an assessment of this kind.

The integration server takes information about statistics and accounting from the operating system. On some operating systems, such as Windows and Linux®, rounding can occur because the system calls that are used to determine the processor times are not sufficiently granular. This rounding might affect the accuracy of the data.

Data relating to the size of messages is not collected for WebSphere® Adapters nodes. For example:
  • The SAPInput node
  • The FileInput node
  • The JMSInput node
  • Any user-defined input node that does not create a message tree from a bit stream.

You can start and stop data collection by modifying the statistics collection properties in the node.conf.yaml or server.conf.yaml configuration file for the integration node or server.

When an integration node or server is created, the publication of snapshot message flow statistics is enabled by default. The publicationOn property in the node.conf.yaml or server.conf.yaml file is explicitly set to active and the outputFormat property is set to json, which enables the publication of snapshot statistics to the web user interface. The publication of archive statistics is turned off by default.

For integration nodes and servers that were created prior to IBM® App Connect Enterprise Version 11.0.0.8, the publication of all message flow statistics (snapshot and archive) was turned off by default. To enable the publication of statistics for integration nodes or servers that were created prior to V11.0.0.8, edit the relevant .conf.yaml file and activate the publicationOn and outputFormat properties.

For more information, see Configuring the collection of message flow statistics by using a .yaml configuration file and Configuring an integration server by modifying the server.conf.yaml file.

You can activate data collection on your production and test systems. If you collect the default level of statistics (message flow), the effect on performance is minimal. However, collecting more data than the default message flow statistics can generate high volumes of report data that might affect performance slightly.

The information in the following topics helps you when you are planning your data collection:

The following topic contains reference information that you might find helpful when you analyze and tune the performance of your message flows: Message flow accounting and statistics records.

Publishing data to IBM MQ and MQTT

If you want to publish data to IBM MQ or MQTT, ensure that the publication of events is enabled and that the pub/sub broker is configured before you start data collection. For more information, see Configuring the publication of event messages and Configuring the built-in MQTT pub/sub broker.

The topic for each message has the following structure:
  • For XML format:
    • For an IBM MQ pub/sub broker:
      $SYS/Broker/integrationNodeName/StatisticsAccounting/integration_server_name
    • For an MQTT pub/sub broker:
      IBM/IntegrationBus/integrationNodeName/StatisticsAccounting/integration_server_name
  • For JSON format:
    • For an IBM MQ pub/sub broker:
      $SYS/Broker/integrationNodeName/Statistics/JSON/integration_server_name
    • For an MQTT pub/sub broker:
      IBM/IntegrationBus/integrationNodeName/Statistics/JSON/integration_server_name
You can set up subscriptions for a specific integration server on a specific integration node. For example:
  • For XML format:
    • For an IBM MQ pub/sub broker:
      $SYS/Broker/INODE/StatisticsAccounting/default
    • For an MQTT pub/sub broker:
      IBM/IntegrationBus/INODE/StatisticsAccounting/default
  • For JSON format:
    • For an IBM MQ pub/sub broker:
      $SYS/Broker/INODE/Statistics/JSON/default
    • For an MQTT pub/sub broker:
      IBM/IntegrationBus/INODE/Statistics/JSON/default
You can also use wildcards in the subscriptions to broaden the scope of what is returned. For example, to subscribe to reports for all integration servers on all integration nodes, use the following values:
  • For XML format:
    • For an IBM MQ pub/sub broker:
      $SYS/Broker/+/StatisticsAccounting/#
    • For an MQTT pub/sub broker:
      IBM/IntegrationBus/+/StatisticsAccounting/#
  • For JSON format:
    • For an IBM MQ pub/sub broker:
      $SYS/Broker/+/Statistics/JSON/#
    • For an MQTT pub/sub broker:
      IBM/IntegrationBus/+/Statistics/JSON/#