Failed to connect to the JMX port on server

Body

Failed to connect to the JMX port on server

When you first connect from MDM Workbench to WebSphere Application Server (AppServer) where MDM Server is installed, for example to deploy a configuration project or to run a virtual job, you might see this error:

  Job Manager Error - Failed to connect to the JMX port on server

There can be several reasons why the connection might fail, for background, here is the stack you are relying on when you connect to the JMX port.

In order for the JMX port connection to be successful, you need every component in this diagram to be in a fully functioning healthy state. And yes, that means there are a lot of places you can check! As a result, it's not practical here to explain every possible area to review, but this should give you some idea of where to start investigating.

To begin, cut the problem in half: there is a message associated with blueprint virtual bridge. Look for this, and it will help you decide whether the problem is more likely to be a runtime issue (below and to the right of the blueprint virtual bridge component) or a configuration issue

1. Look for virtual bridge messages

On the Application Server where MDM is hosted, open SystemOut.log or HPEL logs: if possible restart the AppServer first to make sure you have startup messages:

Success scenario

When the MBean starts successfully, you will see messages like these:

•   RMIConnectorC A ADMC0026I: The RMI Connector is available at port <xxxx>

•   JMXConnectors I ADMC0058I: The JMX JSR160RMI connector is available at port <xxxx>

•   BlueprintCont I org.apache.aries.blueprint.container.BlueprintContainerImpl doRun Bundle com.ibm.mdm.server.virtualbridge is waiting for dependencies [(objectClass=com.ibm.mdm.server.config.api.ConfigManager), (objectClass=com.initiatesystems.hub.mpinet.MPINetProtocolLogic)]

Note that these messages will only appear on startup, so they may not be visible if the logs have wrapped

If you have these success messages the Blueprint virtual bridge is available for JMX requests, and everything to the right of the diagram (MPIJNI, JMS, databases, filesystems) is healthy.

In this case the likely cause of the problems is to the left of the diagram, and probably relates to a configuration issue. More information is available in section 3. When the virtual bridge has started successfully

Failure scenario

When the MBean has not started, you see messages like this:

•   BlueprintCont E org.apache.aries.blueprint.container.BlueprintContainerImpl$1 run Unable to start blueprint container for bundle com.ibm.mdm.server.virtualbridge due to unresolved dependencies [(objectClass=com.ibm.mdm.server.config.api.ConfigManager), (objectClass=com.initiatesystems.hub.mpinet.MPINetProtocolLogic)]

• This message identifies a problem with the JMX MBean listener, but does not in itself identify the root cause: to find the source, look for other messages in the logs with earlier timestamps.

If you have these failure messages the Blueprint virtual bridge is not available. More information is available in the next section, 2. When the virtual bridge has not started

No messages found

If you don't find any messages relating to com.ibm.mdm.server.virtualbridge, the most likely reason is that there were messages when the server started, but the logs have wrapped the virtual bridge messages are no longer current. The recommended action is to restart the server and collect the new logs.

2. When the virtual bridge has not started

When the blueprint virtual bridge has not started, the next step is to investigate potential runtime issues in one or more of the components on the right side of the diagram.

1. Look for Database errors, search the Application Server logs for
   • SQL
   • DSRA
2. Look for JMS errors, the JMX MBean for virtual relies on the SIBus messaging engine, search the Application Server logs for
   • CWSIS

Note that you can choose whether you use a datastore or filestore for the messaging engine data store: the default is datastore (database).

There may be file system errors, these will usually be reported by the component that depends on the file system, for example the database or the JMS filestore.

In many cases you will be able to find technotes or other links on the internet with information about how to resolve the errors, or if not, contact IBM support and provide the logs that show the errors.

These related links have information about resolving blueprint errors:

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W2c3ebf603f05_4460_8e8b_a26780b35b45/page/Troubleshooting%20guidance%20for%20InfoSphere%20MDM%20installations

/support/pages/node/429789

3. When the virtual bridge has started successfully

Once you have found the success message, the next step is to investigate the configuration in both WebSphere Application Server and MDM Workbench.

•   com.ibm.mdm.server.virtualbridge is waiting for dependencies

Review the server logs for authorization errors

On the Application Server where MDM is hosted, open SystemOut.log or HPEL logs. Look for errors that reference one or more of:

• LDAP
• LTPA
• SECJ

Errors with any of these codes suggest that you need to re-visit the security configuration in the WebSphere Application Server administrative console, and check userid and password settings in the workbench client. Review the error messages, in many cases you will be able to find technotes or other links on the internet with information about how to resolve them, or if not, contact IBM support and provide the logs that show them.

Review the firewall settings

Verify that you can ping from the Workbench machine to the machine that hosts WebSphere Application Server and MDM Server, using your preferred ping tool.

Optionally you can use "Test Connection" from MDM Workbench, although note that in an ND configuration this tool only checks the dmgr, so it may not be the correct status for the actual server where MDM is hosted.

If you can not connect to the target MDM server, the JMX connection will not work and you need to contact your networking support team to make sure the network is available and if necessary, that appropriate firewall ports are opened.

Review the port and host configuration

1. In the WebSphere application console
   1. Go to Servers -> Server Types -> WebSphere application servers
   2. Select the server where the MDM runtime is installed
   3. Scroll down to the ports section and open it (see the screenshot below)
   4. Make a note of the ports for
      • BOOTSTRAP_ADDRESS
      • SOAP_CONNECTOR_ADDRESS
      • ORB_LISTENER_ADDRESS (if not 0)
2. In the Workbench
   1. Go to the Servers tab (you may need to add it from Window -> Show View)
   2. In the Overview panel, under Server select Manually provide connection settings
   3. Set the RMI port to BOOTSTRAP_ADDRESS
   4. Set SOAP port to SOAP_CONNECTOR_ADDRESS
   5. Test these settings initially without any IPC port configuration
   6. If you are still not able to connect, also configure the IPC port with the ORB_LISTENER_ADDRESS and retest