Enabling content-based filtering with publish/subscribe

IBM® MQ provides the ability to specify a filter when a subscription is made, but this can only refer to items in headers. IBM App Connect Enterprise can act as a content filtering provider for IBM MQ, and so allows extended filters to be specified by subscribers that can refer to elements in the body of publications.

Before you begin

Read about Selectors on IBM MQ.
Read the following overview of how IBM MQ selects content of messages.
When an application publishes on a topic string, where one or more subscribers have a selection string selecting on the content of the message, IBM MQ requests that the extended message selection provider parse the publication and inform IBM MQ whether the publication matches the selection criteria specified by each subscriber with a content filter.

If the extended message selection provider determines that the publication matches the subscriber's selection string, the message continues to be delivered to the subscriber.

If the extended message selection provider determines that the publication does not match, the message is not delivered to the subscriber. This might cause the IBM MQ MQPUT or MQPUT1 call to fail with reason code MQRC_PUBLICATION_FAILURE. If the extended message selection provider is unable to parse the publication, reason code MQRC_CONTENT_ERROR is returned and the MQPUT or MQPUT1 call fails.

If the extended message selection provider is unavailable or is unable to determine whether the subscriber should receive the publication, reason code MQRC_SELECTION_NOT_AVAILABLE is returned and theIBM MQ MQPUT or MQPUT1 call fails.

When a subscription is being created with a content filter and the extended message selection provider is not available, the IBM MQ MQSUB call fails with reason code MQRC_SELECTION_NOT_AVAILABLE. If a subscription with a content filter is being resumed and the extended message selection provider is not available, the IBM MQ MQSUB call returns a warning of MQRC_SELECTION_NOT_AVAILABLE, but the subscription is allowed to be resumed.
Each queue manager that supports Extended Message Selection (Content Based Filtering), requires a broker that is associated with it and Content Based Filtering enabled on at least one integration server. If a client subscribes and specifies a content-based filtering expression against a queue manager that does not have this function enabled, it receives MQRC_SELECTION_NOT_AVAILABLE on the subscription request.
On Linux®, UNIX and Windows systems, grant authorization for the system to the queue manager:setmqaut -t <qmgr> +system -p <brokerUserId>

About this task

IBM App Connect Enterprise extends the message selection support provided by IBM MQ. IBM App Connect Enterprise does this by allowing ESQL statements rather than SQL92 statements, and filtering based on message content. See Content-based filtering using ESQL for details of the scope and exclusions of supported ESQL.

The main external differences in the current implementation of content-based filtering are:

Content based filtering is no longer limited to IBM MQ MQRFH2 subscribers. IBM App Connect Enterprise provides content filtering services for the following IBM MQ subscribers:
- MQRFH2
- MQSUB
If you are performing content filtering based on the NameValueData field, within an MQRFH2 header, on z/OS®, the data can be present either in the first or second MQRFH2 header. For example, the filter:
```
Root.MQRFH2.mcd.Msd='XML'
```
might not work as you expect on z/OS. Use the following syntax to search all MQRFH2 headers:
```
FOR ANY Root.MQRFH2[] AS I (I.mcd.Msd='XML')
```
IBM MQ message properties are supported as part of the filter expression.
If the publication does not contain an mcd folder, the payload is assumed to be XMLNSC.

Read the following steps to see how to enable content-based filtering on IBM App Connect Enterprise.

Procedure

Set the cbfEnabled property of the ContentBasedFiltering object for the integration server in which you want content-based filtering to run.

Note that you must explicitly enable the cbfEnabled property for content based filtering to work; the default setting is for content based filtering to be off.
Restart the integration server for the change to take effect.
If you enable content based filtering in multiple integration servers on z/OS, content based filtering is active in only one integration server at any time. Subsequent integration servers for which content based filtering is enabled, propagate the following messages to the syslog on start up (for each content based filtering thread) and then every 30 minutes, as they fail to connect to the queue manager:
```
BIP2111E MQ04BRK jheg1 15 IBM App Connect Enterprise INTERNAL ERROR: DIAGNOSTIC INFORMATION 
 'Error occurred in Content Based Filtering Thread'. : ImbCbfWorker(909) 
BIP2624E MQ04BRK jheg1 14 UNABLE TO CONNECT TO QUEUE MANAGER 'MQ04': MQCC=2;
 MQRC=2002; MESSAGE FLOW NODE 'ContentBasedFiltering'. : ImbCbfWorker(214)
```
If you stop the integration server that is currently providing content based filtering services, another integration server for which content based filtering is enabled connects to the queue manager and provides content based filtering services.
Both the evaluationThreads and validationThreads properties default to one if content-based filtering is enabled.

Evaluation threads are used to validate content filters against a given publication at publication time. A network with a high number of publications might require the evaluationThreads property to be increased (up to a maximum of 32) to handle the workload at publication time.

Validation threads are used to validate syntax of content filters at subscription time. A publish/subscribe network with a high number of subscribers, especially if dynamic subscribers, might require the validationThreads property to be increased (up to a maximum of 32) to handle the high throughput of subscription requests.

It is possible to enable this function in multiple integration servers, but you must ensure that any message sets required to parse any published message (and referenced in the mcd folder of that message) are deployed to all the integration servers that have been enabled for content-based filtering.

Any errors encountered parsing the message within the evaluation thread cause IBM MQ to return MQRC_CONTENT_ERROR to the publishing application. The parsing error appears as well in the event log as an IBM App Connect Enterprise exception.
Example of a subscription <psc> folder processing ESQL and message properties in the filter:
```
<psc>
<Topic>topic</Topic>
   <Filter>
   SUBSTRING(Root.XMLNSC.Name.FirstName FROM 1 FOR 1) = ‘J' and usr.flag = ‘yes'
   </Filter>
</psc>
```