Here is the story:
Customer X migrated successfully to DB2 10 CM but observed very significant elapsed time degradation for some batch jobs and DB2 utility jobs. They also noticed that the DB2 prefetch engines were all in use and saw a very high count for 'prefetch engine not available' in the DB2 Statistics Trace. They have one zIIP engine on each LPAR and it is maxed out most of the time. Their immediate tuning action was to take the zIIP engine away from one of the LPARs, and guess what? All the elapsed time degradation problems went away on that LPAR.
So what was happening and how did they resolve the problem?
In DB2 10, prefetch and deferred writes engines are now eligible for zIIP offload - but they still need to be dispatched very quickly. Any delays could result in all of the above happening.
Customer X was running with all the defaults in parmlib member IEAOPTxx:
HIPERDISPATCH=YES (default on z196 hardware - and recommended best practice)
IIPHONORPRIORITY=YES (default - and recommended best practice), which means that zIIP processors can get help from standard processors (GCP)
ZIIPAWMT=3200 (default when HIPERDISPATCH=YES), which means that the zIIP needs to be running for 3.2 msec before it checks to see if it should ask for help from GCPs (this is called 'alternate wait management'). Many requests can be flowing through the zIIP during this time period. But if the zIIP has been running for the length of time specified by ZIIPAWMT, the queue of work is still growing, and all the zIIPs are busy, then the zIIP signals a GCP for help to process the work.
With the above default settings, if the zIIP capacity is under-configured, the DB2 prefetch engines can end up queuing for up to 3.2 msec for a zIIP before they are dispatched on a GCP. Of course, this could be much worse if the zIIP processors were not allowed to ask GCPs for help (IIPHONORPRIORITY=NO).
Now, what follows is NOT meant to sound like a sales pitch, but the correct technical solution is to add more zIIPs. It would be prudent to have the zIIPs run in the 30-50% CPU busy range on average (peaks would obviously be higher). zIIPs should be thought of as assist processors and they are not intended to be run as hard as GCPs. This recommendation is not specific to DB2 10, but with DB2 10, it has become even more critical. This is the option that Customer X decided to take - they have now refreshed their processor technology and increased the ratio of zIIP engines to GCP engines. Consequently the original elapsed time performance problems have gone away.