We've seen some interesting issues come up over the last several months. Many of these were related to security on all platforms. On z/OS some issues involving ICSF came to light this year. ICSF is used on the z/platform, among other things, to provide entropy that improves password protection. In particular users had experienced issues with version 8 clients that connect using the MQ Explorer where password protection was being invoked. To level set, when password protection is attempted, at version 8 MQ will attempt to negotiate it with the client if three conditions are true: (1) The channel involved is a SVRCONN, (2) there is no SSL/TLS encryption defined for that channel, and (3) the level of protocol used on the channel meets FAP level 13 or higher. Also, on the Explorer side the 'user identification compatibility mode' attribute must not be enabled. If the channel used has a non-null cipherspec specified, then password negotiation would not be attempted since any passwords on it would not pass as plaintext anyway. You can review the list of required conditions in the product documentation.
When password protection fails during connection negotiation, CSQX296E is generated. Since ICSF is used by MQ to generate random data, used by password protection, this means that READ authority needs to be granted to the STC userid in the CSFSERV profile. When trying to determine if ICSF may be at issue, the MQ joblog should be checked for CSQX213E. That message will provide a return code and reason. If the RC is 12, that indicates ICSF is inactive. The reason is documented in the z/OS Crypto ICSF Application Programmer's Guide.
The ICSF function is an installable hardware feature that allows the ICSF region to be started in order to provide services. However, ICSF services can be disabled, in that case MQ will fallback to use the operating system STCK function instead. If ICSF is used, then the callable service, CSFPPRF needs to be available. Users should ensure that the fix for PI53111 is installed. If the CHIN has not been allowed sufficient access to the CSFRNG resource (type CSFSERV), then MQ will likely generate errors to the CHIN joblog similar to CSQX571E where the callable service will be flagged. The messages for the external security manager (RACF, Top Secret, ACF2, etc.) should also be checked for any anomalies. Note that password protection is only particular to MQ v8 (so the configuration used at version 7 may no longer be appropriate if password protection is expected to function properly after migration). This means certain checks by MQ regarding SSL were not done at version 7, but are done at version 8, thus requiring MQ to have a PERMIT to resource TYPE=CSFSERV which was not necessary before.
Then there are the reported cases of MQRC 2141 being returned by MQ for the Dead Letter Queue Handler (after some users migrate to version 8). While PI28005 documents a defect issue where the CodeCharSetID is not set properly (in a scenario associated with this MQRC), receipt of MQRC 2141 really means that the message that is being sent to the dead letter queue is, in some way, improperly formed. In IBM MQ version 8, if a message is put with MQMD and format is set to 'MQDEAD', then additional checks are now done to ensure that such a message really does start with an MQDLH; otherwise it's failed as described above. This point is worth keeping in mind at migration time since any applications affected by this policing may need to be modified so that the message format is not set to 'MQDEAD' or that the message data begins with an MQDLH.
WAS and Java/JMS
Also seen after migration to IBM MQ v8 (but related to Java/JMS) is receipt of error message JVMCFRE003. Applications that worked fine at v7 now fail because of the Java version being used. WebSphere MQ classes for JMS v8 implements the JMS 2.0 specification (which also required Java 1.7). Because it is not allowed to use Java classes on an earlier version of the JRE runtime than what they were compiled with, the above error will be returned. The solution is to use Java 1.7 (version 7) or later to run the application under. Conversely, WebSphere Application Server (WAS) 8.0 does NOT use the WebSphere MQ Resource Adapter v8 (which contains the same WMQ-JMS classes being used). The WMQ-JMS classes are compatible with all supported versions of queue manager (so no synchronization is needed) allowing WebSphere Application Server to communicate with a v8 queue manager, without issue.
MQ users that employ CHLAUTH rules want to be aware of the IBM MQ on z/OS HIPER APAR PI66960. This open (at the time this blog entry was written) APAR documents that DISPLAY CHLAUTH (with the RUNCHECK option) can cause an abend EC6-0F01C008 against the address space of the user issuing the command. This can occur if there is no OMVS segment defined for the queue manager, since, at v8 MQ, CHLAUTH processing invokes a DNS reverse lookup request. This makes use of USS services (thus the need for the OMVS segment with a UID to exist). As no PTF is currently available (at the time this blog entry was written), this error is avoided by disabling reverse lookups; ie. ALTER QMGR REVDNS(DISABLED).
MQ on z/OS hangs
The new HIPER APAR PI66686 (mentioned above) in conjunction with zOS maintenance that's received through PUT 1604 or RSU 1604 should also be on your radar. Depending on your level of z/OS, if either UA81297, UA81298, or UA81300 has been applied there's the chance termination of applications or the queue manager will end in a hang while waiting for tasks to end. Either the MQ PTF UI40126 can be applied to remediate the hang or, alternatively, until that fix can be applied, the z/OS maintenance can be restored. OA49676 introduced the issue which PI66686 fixes. Users of WAS would find that the server will not end normally, or shutdown of the control region may time out with ABENDSA03 and ABENDSE6C. z/OS APAR OA50970 is created to fix the issue introduced by OA49676 from the operating system's perspective.
Importance of adhering to dataset SHAREOPTIONS
Certain z/OS datasets within MQ mandate that SHAREOPTIONS(2,3) be set and adhered to. This requirement includes the log data sets and the BSDS which tracks the inventory of MQ's logs. The default JCL (CSQ4BSDS) shipped with MQ defines these value with SHAREOPTIONS(2,3). This setting ensures that any number of opens against the data set is allowed for reading, but only one region can update the bootstrap at any one time. If a user were to set SHAREOPTIONS(3,3) this would mean all regions can read and write to/from the data set at any time, possibly affecting the integrity of the data set. In this second case we see that users who also use the CSQJU003 utility to update the BSDS (while the queue manager is running) can cause pages within the BSDS to become orphaned that may later prevent MQ from adding additional archive log entries to that data set. This violates two documented requirements (1) that the SHAREOPTIONS(2,3) be used, and (2) that any utility similar to CSQJU003 only be run when the queue manager is inactive. While it could take years for such corruption to manifest itself, receipt of CSQJ126E (without a preceding instance of CSQJ107E or CSQJ108E) may be evidence that this utility has been improperly used. Note that the now unavailable SupportPac MS15 is subject to the same usage requirements. In the worst case, recovery would mean a cold start of the queue manager so best practice is to use the default SHAREOPTIONS provided while invoking these utilities only in standalone mode.