Do you use channel exits written in C on your z/OS queue manager?
tonySharkey 060000E7M1 Comments (4) Visits (4032)
There are multiple types of channel exits supported by MQ, namely SENDEXIT, RCVEXIT, SCYEXIT and MSGEXITs. It is possible to have one or more of each of these associated with any particular channel, except MSGEXIT for either SVRCONN or CLNTCONN-type channels.
When a channel exit written in C is invoked, the Language Environment needs to be built. One of the first events to take place is that z/OS loads the C runtime event handler module CEEEV003.
Depending on where this module is loaded from can make a significant difference to the time spent processing the exit. If the module needs to be read from DASD each time, performance could be improved.
APAR PI58641 may improve the performance of your C channel exits by pre-loading the CEEEV003 module, which will bypass the repeated DASD I/O, which as well as improving the time to process the exit call can also reduce the load on the DASD volume hosting the dataset containing the CEEEV003 module.
In our measurements we saw up to 16 times increase in message throughput to tests with send and message exits once the APAR was applied but a more typical improvement was of the order 3 times.
Is there a simple way to see if I will benefit from the APAR?
If the CEEEV003 module is not in the LPALST, it is less likely that you will benefit from the APAR as when the MQ channel initiator starts its name server as part of initialisation, CEEEV003 will be loaded and this will result in the module being placed in the Job Pack Area (JPA) and this will mean that subsequent loads can be satisfied without accessing the DASD copy of the module.
This does mean that every address space that requires CEEEV003 will have to load the module into their own JPA, resulting in multiple copies of the module.
When the CEEEV003 module is found in the LPALST, the channel initiators' STEPLIB can be temporarily altered to have the SCEERUN library in the concatenation and restarted. Once the channel initiator name server starts, the module will be loaded into the JPA reducing the subsequent load times.
Note: Once the APAR is applied, remove the SCEERUN dataset from the STEPLIB.
Where am I picking up CEEEV003 from?
The ISPF application “TSO ISRFIND” can be used to see where the module is being found e.g.
Note: Select “Y” for LOADMOD, to ensure that the LPA / LPALIST and LINKLST is checked.
We found copies in:
How long are my exits taking to run?
The “DISPLAY CHSTATUS(*) EXITTIME” command can be used to determine how long the exit is taking, but this is not the complete picture as it is the total elapsed time spent in exit processing per message, which includes the time to load the Language Environment module.
Can MQ V8.0 give me more information?
In MQ V8.0, channel initiator statistics were added which can be used to monitor the time taken by each dispatcher task.
An example of the dispatcher report as produced by MQSMF, which is available as part of the MP1B supportPac, is shown here:
In this example there are 2 busy channels sharing the same dispatcher task.
But I don't use MQ V8.0 channel statistics, so how else can I see if there is a problem?
A dump of the channel initiator would show how long the load of the environment is taking.
Formatting the dump in IPCS using “SYSTRACE TIME(LOCAL)” gives output similar to:
In this example, the “SVC 8” (load) can be checked using IPCS command ‘IP L 2225986C’ (unique-2) and this shows the eye-catcher CEEEV003.
There will be a corresponding “SVCR 8” (unload) one the use of the module is complete. If the module is not already loaded in either the Job Pack Area (JPA), Link Pack Area (LPA), it may need to be read from DASD, which will be a large factor in how much time there is between the load and unload calls.
Why is this important?
When the MQ channel initiators dispatcher task is loading the exit, it is unable to process work for other channels. Since a channel has an affiliation to a dispatcher task for its life-time, one channel effectively blocking the dispatcher task while waiting for repeated channel exits can cause an impact on the other channels allocated to the same dispatcher task.
How much of an improvement might I see?
For simple measurements where we were only running channels with channel exits configured, we saw up to 3 times higher throughput with the APAR applied with a cost reduction of up to 50% in the channel initiator address space.
When running with multiple exits, the improvement was even more marked. Transaction cost in the channel initiator address space dropped by 60% and transaction rate improved 6 times when the APAR was applied as the dispatcher task was no longer constrained when processing a single channel.
The following 2 charts show the improvement in transaction rate and cost when the APAR is applied and how that compares to the same workload when the channels do not have a SENDEXIT configured.
These 2 charts show a significant improvement in performance with the APAR applied, such that transaction cost and rate are close to the performance of the measurements where no exits are configured.
We found that as the workload increased to use more channels, the impact of APAR PI58641 was greater both in improving the throughput of the channels with exits and in reducing the load on the dispatcher, thereby allowing other channels to use the same dispatcher resource.
Note: The send exit used performs little logic beyond checking the validity of the data.
Using the MQ V8.0 channel initiator statistics data for the dispatchers:
In this set of data, the 2 busy channels are running on separate dispatcher tasks.
Finally we also see a decrease in the EXITTIME as reported by the "DISPLAY CHSTATUS(*) EXITTIME" command from 800 to 26 microseconds.