JMS connectivity issues by using WebSphere Application Server bundled with IBM MQ

JMS connection for Product Master is set configured by using WebSphere® Application Server bundled with IBM® MQ messaging provider.

Symptoms

Using Product Master script sandbox user interface, JMS connection can be established but the same script code snippet when used in a report job that runs under scheduler service, fails with exceptions.

Environment

The following exceptions are examples of what you might see:
2013-06-26 13:12:01,533 [sch_worker_0] ERROR com.ibm.ccd.common.error.AustinException JOB_ID:2091- CWPCM0002E:Generic error / Exception: Generic Error, Exception:Failed to create InitialContext using factory specified in hashtable javax.naming.NoInitialContextException: Failed to create InitialContext using factory specified in hashtable [Root exception is java.lang.NullPointerException] at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:255) at javax.naming.InitialContext.initializeDefaultInitCtx(InitialContext.java:318) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:348) at javax.naming.InitialContext.internalInit(InitialContext.java:286) at javax.naming.InitialContext.(InitialContext.java:211) at com.ibm.ccd.connectivity.jms.TrigoJMS.getInitialContext(TrigoJMS.java:88) at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetContext(ScriptOperationsJms.java:42) at com.ibm.ccd.common.interpreter.operation.generated.GenJmsGetContextOperation.execute(GenJmsGetContextOperation.java:74) at WPCLGReport13722406649020.run(WPCLGReport13722406649020.java:37) at com.ibm.ccd.common.interpreter.engine.Script.runFunction(Script.java:554) at com.ibm.ccd.common.interpreter.engine.Script.execute(Script.java:484) at com.ibm.ccd.common.interpreter.engine.Script.run(Script.java:335) at com.ibm.ccd.report.common.Report.generate(Report.java:305) at com.ibm.ccd.report.common.ReportExe.execute(ReportExe.java:104) at com.ibm.ccd.scheduler.threads.SchedulerThread.fuzaoRun(SchedulerThread.java:262) at com.ibm.ccd.common.util.FuzaoRunnableAdapter.run(FuzaoRunnableAdapter.java:54) at com.ibm.ccd.common.util.FuzaoThread.run(FuzaoThread.java:123) Caused by: java.lang.NullPointerException at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:247) ... 16 more 2013-06-26 13:12:01,534 [sch_worker_0] ERROR com.ibm.ccd.common.error.AustinException JOB_ID:2091- CWPCM0002E:Generic error / Exception: Script execution failed (com.ibm.ccd.common.error.AustinException) Exception:Generic Error Generic Error at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetContext(ScriptOperationsJms.java:46) at com.ibm.ccd.common.interpreter.operation.generated.GenJmsGetContextOperation.execute(GenJmsGetContextOperation.java:74) at WPCLGReport13722406649020.run(WPCLGReport13722406649020.java:37) at com.ibm.ccd.common.interpreter.engine.Script.runFunction(Script.java:554) at com.ibm.ccd.common.interpreter.engine.Script.execute(Script.java:484) at com.ibm.ccd.common.interpreter.engine.Script.run(Script.java:335) at com.ibm.ccd.report.common.Report.generate(Report.java:305) at com.ibm.ccd.report.common.ReportExe.execute(ReportExe.java:104) at com.ibm.ccd.scheduler.threads.SchedulerThread.fuzaoRun(SchedulerThread.java:262) at com.ibm.ccd.common.util.FuzaoRunnableAdapter.run(FuzaoRunnableAdapter.java:54) at com.ibm.ccd.common.util.FuzaoThread.run(FuzaoThread.java:123) Caused by: javax.naming.NoInitialContextException: Failed to create InitialContext using factory specified in hashtable [Root exception is java.lang.NullPointerException] at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:255) at javax.naming.InitialContext.initializeDefaultInitCtx(InitialContext.java:318) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:348) at javax.naming.InitialContext.internalInit(InitialContext.java:286) at javax.naming.InitialContext.(InitialContext.java:211) at com.ibm.ccd.connectivity.jms.TrigoJMS.getInitialContext(TrigoJMS.java:88) at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetContext(ScriptOperationsJms.java:42) ... 10 more 2013-07-17 10:36:48,401 [sch_worker_1] ERROR com.ibm.ccd.common.error.AustinException JOB_ID:6203-CWPCM0002E:Generic error / Exception: Generic Error, Exception: javax.naming.Reference incompatible with javax.jms.QueueConnectionFactory java.lang.ClassCastException: javax.naming.Reference incompatible with javax.jms.QueueConnectionFactory at com.ibm.ccd.connectivity.jms.TrigoJMS.getConnectionFactory(TrigoJMS.java:104) at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetConnectionFactory(ScriptOperationsJms.java:57) at com.ibm.ccd.common.interpreter.operation.generated.GenJmsGetConnectionFactoryOperation.execute(GenJmsGetConnectionFactoryOperation.java:74) at WPCJms13740502073320.run(WPCJms13740502073320.java:39) at com.ibm.ccd.common.interpreter.engine.Script.runFunction(Script.java:554) at com.ibm.ccd.common.interpreter.engine.Script.execute(Script.java:484) at com.ibm.ccd.common.interpreter.engine.Script.run(Script.java:335) at com.ibm.ccd.report.common.Report.generate(Report.java:305) at com.ibm.ccd.report.common.ReportExe.execute(ReportExe.java:104) at com.ibm.ccd.scheduler.threads.SchedulerThread.fuzaoRun(SchedulerThread.java:262) at com.ibm.ccd.common.util.FuzaoRunnableAdapter.run(FuzaoRunnableAdapter.java:54) at com.ibm.ccd.common.util.FuzaoThread.run(FuzaoThread.java:123) 2013-07-17 10:36:48,401 [sch_worker_1] ERROR com.ibm.ccd.common.error.AustinException JOB_ID:6203-CWPCM0002E:Generic error / Exception: Script execution failed(com.ibm.ccd.common.error.AustinException) Exception:Generic Error Generic Error at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetConnectionFactory(ScriptOperationsJms.java:61) at com.ibm.ccd.common.interpreter.operation.generated.GenJmsGetConnectionFactoryOperation.execute(GenJmsGetConnectionFactoryOperation.java:74) at WPCJms13740502073320.run(WPCJms13740502073320.java:39) at com.ibm.ccd.common.interpreter.engine.Script.runFunction(Script.java:554) at com.ibm.ccd.common.interpreter.engine.Script.execute(Script.java:484) at com.ibm.ccd.common.interpreter.engine.Script.run(Script.java:335) at com.ibm.ccd.report.common.Report.generate(Report.java:305) at com.ibm.ccd.report.common.ReportExe.execute(ReportExe.java:104) at com.ibm.ccd.scheduler.threads.SchedulerThread.fuzaoRun(SchedulerThread.java:262) at com.ibm.ccd.common.util.FuzaoRunnableAdapter.run(FuzaoRunnableAdapter.java:54) at com.ibm.ccd.common.util.FuzaoThread.run(FuzaoThread.java:123) Caused by: java.lang.ClassCastException: javax.naming.Reference incompatible with javax.jms.QueueConnectionFactory at com.ibm.ccd.connectivity.jms.TrigoJMS.getConnectionFactory(TrigoJMS.java:104) at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetConnectionFactory(ScriptOperationsJms.java:57) ... 10 more , Exception:Generic Error Generic Error at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetConnectionFactory(ScriptOperationsJms.java:61) at com.ibm.ccd.common.interpreter.operation.generated.GenJmsGetConnectionFactoryOperation.execute(GenJmsGetConnectionFactoryOperation.java:74) at WPCJms13740502073320.run(WPCJms13740502073320.java:39) at com.ibm.ccd.common.interpreter.engine.Script.runFunction(Script.java:554) at com.ibm.ccd.common.interpreter.engine.Script.execute(Script.java:484) at com.ibm.ccd.common.interpreter.engine.Script.run(Script.java:335) at com.ibm.ccd.report.common.Report.generate(Report.java:305) at com.ibm.ccd.report.common.ReportExe.execute(ReportExe.java:104) at com.ibm.ccd.scheduler.threads.SchedulerThread.fuzaoRun(SchedulerThread.java:262) at com.ibm.ccd.common.util.FuzaoRunnableAdapter.run(FuzaoRunnableAdapter.java:54) at com.ibm.ccd.common.util.FuzaoThread.run(FuzaoThread.java:123) Caused by: java.lang.ClassCastException: javax.naming.Reference incompatible with javax.jms.QueueConnectionFactory at com.ibm.ccd.connectivity.jms.TrigoJMS.getConnectionFactory(TrigoJMS.java:104) at com.ibm.ccd.common.script.ScriptOperationsJms.jmsGetConnectionFactory(ScriptOperationsJms.java:57) ... 10 more

Resolving the problem

JMS jars that are bundled in WebSphere Application Server are used by WebSphere Application Server runtime and are provisioned for services that are started within WebSphere Application Server boundaries (contained by WebSphere Application Server). When using a stand-alone client, process, or service for creating a JMS connection that is maintained by WebSphere Application Server, these jars are required to be included in the class path of the JVM for that client, process, or service. In this case, it is Product Master scheduler service. Therefore, to solve this issue, the following WebSphere Application Server bundled runtime jars are required to be added in the class path for the scheduler service.
  • For WebSphere Application Server 8.5.x:
    • $WAS_HOME/runtimes/com.ibm.ws.admin.client_8.5.0.jar
    • $WAS_HOME/runtimes/com.ibm.jaxws.thinclient_8.5.0.jar
    • $WAS_HOME/runtimes/com.ibm.ws.sib.client.thin.jms_8.5.0.jar
The following additional JAR files are also required:
  • $WAS_HOME/plugins/com.ibm.ws.runtime.jar
  • $WAS_HOME/plugins/com.ibm.ws.sib.server.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.commonservices.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.connector.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.headers.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.local.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.remote.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.system.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jms.admin.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mq.pcf.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.mqjms.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.commonservices.j2se.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.commonservices.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.jms.internal.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.jms.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.matchspace.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.provider.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.ref.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.common.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.factories.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.v6.jar
  • $WAS_HOME/installedConnectors/wmq.jmsra.rar/dhbcore.jar
Note: Under the classpath parameter in the $TOP/bin/conf/env_settings.ini directory, the Product Master appserver will fail to start as inclusion of these jars in the classpath causes issues in the WebSphere Application Server class path. Therefore, the following steps are recommended.
Note: Anytime Product Master is upgraded with a new maintenance patch, these steps are required.
  1. Stop all of the Product Master services by using the stop script:
    cd $TOP/bin/go
      ./abort_local.sh
  2. Edit the $TOP/bin/conf/env_settings.ini file to add the WebSphere Application Server JMS runtime jars towards the end of the classpath variable separated by ":", for example:
    classpath=<Existing_Classpath_Entries>:/opt/IBM/WebSphere/AppServer/runtimes/com.ibm.ws.admin.client_7.0.0.jar:/opt/IBM/WebSphere/AppServer/runtimes/com.ibm.jaxws.thinclient_7.0.0.jar:/opt/IBM/WebSphere/AppServer/runtimes/com.ibm.ws.sib.client.thin.jms_7.0.0.jar:/opt/IBM/WebSphere/AppServer/plugins/com.ibm.ws.runtime.jar:/opt/IBM/WebSphere/AppServer/plugins/com.ibm.ws.sib.server.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.commonservices.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.connector.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.headers.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.local.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.remote.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jmqi.system.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.jms.admin.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mq.pcf.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.mqjms.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.commonservices.j2se.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.commonservices.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.jms.internal.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.jms.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.matchspace.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.provider.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.ref.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.common.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.factories.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/com.ibm.msg.client.wmq.v6.jar:/opt/IBM/WebSphere/AppServer/installedConnectors/wmq.jmsra.rar/dhbcore.jar
  3. Start all of the Product Master services by using the start script:
    cd $TOP/bin/go
      ./start_local_rmlogs.sh
    Note: You can need to back up to the $TOP/logs directory.
    Note: Product Master appserver process fails to start and this is expected. This is because of the inclusion of WebSphere Application Server runtime jars in the class path that gets updated in WebSphere Application Server for Product Master appserver. Therefore, the next step should be to remove these WebSphere Application Server JMS runtime jars from deployed Product Master appserver in WebSphere Application Server.
  4. Start the default server "server1" of WebSphere Application Server, if not already started:
    cd <WAS_HOME>/profiles/AppSrv01/bin
      ./startServer.sh server1
  5.  Log in to he WebSphere Application Server admin console:
    <http://<ServerName_Or_IP>:9060/ibm/console/login.do>
    where 9060 is your console port number.
  6. On left side navigation, click Servers > Server Types > WebSphere Application Servers.
  7. Open the server where Product Master is installed, as defined in the $TOP/bin/conf/env_settings.ini file. Click Java and Process Management > Process definition > Java Virtual Machine.
  8. Remove the entries for the WebSphere Application Server JMS runtime jars from the Classpath property. The added jars are towards the end of the class path entry.
  9. Click Apply and then Save.
  10. Stop all of the Product Master services by using the stop script:
    cd $TOP/bin/go
        ./abort_local.sh
        
  11. Start all of the Product Master services by using the start script:
    cd $TOP/bin/go
      ./start_local_rmlogs.sh
    Note: You can need to back up to the $TOP/logs directory.

Now all of the services should start normally. You can now restart all Product Master services by using the default out of the box scripts, unless there is a redeployment that is done by using the install_war.sh file for moving to a higher fix pack or interim fix, in which case the WebSphere Application Server class path would need to be modified manually as detailed preceding section.