SI not working on DR system, activeMQ fails to start

Symptom

After switching to DR environment SI does not come up again.

activemqBroker.log

args[0]=/sterling/install/activemq/conf/activemqconfig.xml
args[1]=activemq.txt
log4j:WARN No appenders could be found for logger (org.apache.xbean.spring.context.ResourceXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.
******Fail to start activemq
activemq startup is defined as Mandatory=true in activeMQ.prop
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in Config File [/sterling/install/activemq/conf/activemqconfig.xml]: Invocation of init method failed; nested exception is java.io.IOException: Cannot create PoolableConnectionFactory (Io exception: The Network Adapter could not establish the connection)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.ini tializeBean(AbstractAutowireCapableBeanFactory.java:1337)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doC reateBean(AbstractAutowireCapableBeanFactory.java:473)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.r un(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(AccessController.java:219)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.cre ateBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(Abstra ctBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingle ton(DefaultSingletonBeanRegistry.java:221)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(Abstract BeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBe anFactory.java:185)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBe anFactory.java:164)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstanti ateSingletons(DefaultListableBeanFactory.java:429)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactory Initialization(AbstractApplicationContext.java:729)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractA pplicationContext.java:381)
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXml ApplicationContext.java:64)
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXml ApplicationContext.java:52)
at com.sterlingcommerce.jms.activemq.SCIBrokerFactory.createApplicationContext(SCIB rokerFactory.java:141)
at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.jav a:52)
at com.sterlingcommerce.jms.activemq.SCIBrokerFactory.main(SCIBrokerFactory.java:64 )
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)<br /> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav a:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at com.sterlingcommerce.woodstock.noapp.NoAppLoader.main(NoAppLoader.java:81)
Caused by: java.io.IOException: Cannot create PoolableConnectionFactory (Io exception: The Network Adapter could not establish the connection)
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)<b r /> at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionConte xt.java:61)
at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.createAdapter(JDBCPersiste nceAdapter.java:317)
at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getAdapter(JDBCPersistence Adapter.java:261)
at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapt er.java:157)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:458)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerServi ce.java:52)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.inv okeInitMethods(AbstractAutowireCapableBeanFactory.java:1368)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.ini tializeBean(AbstractAutowireCapableBeanFactory.java:1334)
... 22 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Io exception: The Network Adapter could not establish the connection)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:85 9)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)< br /> at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionConte xt.java:54)
... 29 more
Caused by: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)< br /> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectio nFactory.java:37)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionF actory.java:290)
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSourc e.java:881)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:85 5)
... 31 more
ACTIVEMQ_HOME: /sterling/install/jar/activemq
ACTIVEMQ_BASE: /sterling/install/jar/activemq
ERROR: java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: Thepst00; nested exception is:
java.net.ConnectException: Connection refused]
java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: Thepst00; nested exception is:
java.net.ConnectException: Connection refused]
at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java :106)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java :57)
at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxComman d.java:224)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:129)< br /> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java :57)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)<br /> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav a:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.apache.activemq.console.Main.runTaskClass(Main.java:225)
at org.apache.activemq.console.Main.main(Main.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)<br /> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav a:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at com.sterlingcommerce.woodstock.noapp.NoAppLoader.main(NoAppLoader.java:81)
ERROR: java.lang.Exception: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: Thepst00; nested exception is:
java.net.ConnectException: Connection refused]
java.lang.Exception: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: Thepst00; nested exception is:
java.net.ConnectException: Connection refused]
at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java :107)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java :57)
at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxComman d.java:224)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:129)< br /> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java :57)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)<br /> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav a:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.apache.activemq.console.Main.runTaskClass(Main.java:225)
at org.apache.activemq.console.Main.main(Main.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)<br /> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav a:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at com.sterlingcommerce.woodstock.noapp.NoAppLoader.main(NoAppLoader.java:81)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: Thepst00; nested exception is:
java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:263 )
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:222 )
at org.apache.activemq.console.command.AbstractJmxCommand.createJmxConnector(Abstra ctJmxCommand.java:146)
at org.apache.activemq.console.command.AbstractJmxCommand.createJmxConnection(Abstr actJmxCommand.java:169)
at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java :70)
... 16 more
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: Thepst00; nested exception is:
java.net.ConnectException: Connection refused]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:112)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:200)<br /> at javax.naming.InitialContext.lookup(InitialContext.java:363)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:184 8)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1818)<b r /> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:284)
... 21 more
Caused by: java.rmi.ConnectException: Connection refused to host: Thepst00; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:590)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:204)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:190)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:321)
at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:89)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:108)
... 26 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:391)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:252)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:239)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:389)
at java.net.Socket.connect(Socket.java:551)
at java.net.Socket.connect(Socket.java:501)
at java.net.Socket.<init>(Socket.java:409)
at java.net.Socket.<init>(Socket.java:223)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFacto ry.java:41)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFacto ry.java:145)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:585)
... 31 more
Adding these pids to the kill list:
All pids