Topic
  • 19 replies
  • Latest Post - ‏2014-10-31T17:45:40Z by RoddiPotter
tomtommoiron
tomtommoiron
8 Posts

Pinned topic Health Center. No JVM detected on localhost on port 1972

‏2011-06-30T09:37:27Z |
I installed the Agent on a AIX WebSphere and the JVM start seems to be correct. It listens on the 1972 port. Validated with netstat.
I installed the ISA HealthCenter tool. I verify with netstat that the connection is possible between the 2 hosts.

But I have a No JVM detected on localhost on port 1972 message on the ISA when I try to connect.

Here an extract of the netstats I perform while trying to connect
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 7 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 7 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 16 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 68 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 68 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 68 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 68 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 324 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED

And what I see in the /tmp/healthcenter.*.log
Jun 30, 2011 10:18:44 AM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgent
INFO: Health Center agent started on port 1972.
Jun 30, 2011 10:18:44 AM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgent
INFO: Agent version "1.3.0.20110107"

I also have an empty /tmp/healthcenter.*.log.lck

Is there a way to add health center logs to ind what is wrong? In which log file should I found more detailed logs?

Thanks for help
Updated on 2012-12-07T12:48:02Z at 2012-12-07T12:48:02Z by SystemAdmin
  • tomtommoiron
    tomtommoiron
    8 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T09:39:49Z  
    Is there a way to add health center logs to find what is the cause of the connection issue? In which log file should I found more detailed logs?
  • tomtommoiron
    tomtommoiron
    8 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T09:40:41Z  
    _
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T09:40:51Z  
    Hi,

    Are you using any sort of port tunnelling to get to the machine what is running the health center agent ?

    regards

    Toby
  • tomtommoiron
    tomtommoiron
    8 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T09:42:23Z  
    Hi,

    Are you using any sort of port tunnelling to get to the machine what is running the health center agent ?

    regards

    Toby
    Yes, I use a ssh tunnel to connect from my ISA (on a windows system) to the WAS JVM (on a AIX System).
  • tomtommoiron
    tomtommoiron
    8 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T10:09:48Z  
    Yes, I use a ssh tunnel to connect from my ISA (on a windows system) to the WAS JVM (on a AIX System).
    I notice a difference between connection attempt when the tunnel is on and when it is off.

    When the tunnel is on, I see a message Searching localhost on port 1972... timeout in xxx seconds and after, I have the No JVM detected on localhost on port 1972 message.
    When the tunnel is off, I directly see the No JVM detected on localhost on port 1972

    The tunnel is from the localhost port 1972 to the AIX host port 1972.
  • tomtommoiron
    tomtommoiron
    8 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T10:13:42Z  
    Yes, I use a ssh tunnel to connect from my ISA (on a windows system) to the WAS JVM (on a AIX System).
    I notice a difference between connection attempt when the tunnel is on and when it is off.

    When the tunnel is on, I see a message Searching localhost on port 1972... timeout in xxx seconds and after, I have the No JVM detected on localhost on port 1972 message.
    When the tunnel is off, I directly see the No JVM detected on localhost on port 1972

    The tunnel is from the localhost port 1972 to the AIX host port 1972.
  • rrosseter
    rrosseter
    2 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-07-05T18:59:30Z  
    I'm having a very similar issue after trying to upgrade from v1.2 to v1.3.

    High level sequence of events.
    • Installed v1.2 late last year which has been working fine until now.
    • Updated client side to HC Agent release v1.3.20110506
    • Connected to JVMs without issue but recieved notice that the agent was out of date (which makes sense as it was the v1.2 HC agent).
    • Used Support Assistant Help module to download latest HC Agent but the files are only dated 1/7/2011.
    • Installed HC Agent by taking backups of all files contained within the agent download and unzipping the zipfile at the correct folder level.
    • Verified that all backed up libraries were present and that new HC agent files were also in place with 01/07/2011 date.
    • Restarted JVMs
    • Can no longer connect to the JVMs using the HC client.
    • Logs in /tmp also show that the HC agents are active (similar messages as tomtommoiron reported)
    • Telnet tests to corresponding ports whre HC agents are supposed to be active are successful from my desktop.

    Configuration: P570 micropartition frames with 48GB memory running on 64bit AIX 5.3.11.5 TL11 with WAS 32bit http://ND 6.1.0.35 cf351044.07 vertically/horizontally clustered across 2 application servers running 4 JVMs each.

    • Rob
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-07-06T08:08:02Z  
    • rrosseter
    • ‏2011-07-05T18:59:30Z
    I'm having a very similar issue after trying to upgrade from v1.2 to v1.3.

    High level sequence of events.
    • Installed v1.2 late last year which has been working fine until now.
    • Updated client side to HC Agent release v1.3.20110506
    • Connected to JVMs without issue but recieved notice that the agent was out of date (which makes sense as it was the v1.2 HC agent).
    • Used Support Assistant Help module to download latest HC Agent but the files are only dated 1/7/2011.
    • Installed HC Agent by taking backups of all files contained within the agent download and unzipping the zipfile at the correct folder level.
    • Verified that all backed up libraries were present and that new HC agent files were also in place with 01/07/2011 date.
    • Restarted JVMs
    • Can no longer connect to the JVMs using the HC client.
    • Logs in /tmp also show that the HC agents are active (similar messages as tomtommoiron reported)
    • Telnet tests to corresponding ports whre HC agents are supposed to be active are successful from my desktop.

    Configuration: P570 micropartition frames with 48GB memory running on 64bit AIX 5.3.11.5 TL11 with WAS 32bit http://ND 6.1.0.35 cf351044.07 vertically/horizontally clustered across 2 application servers running 4 JVMs each.

    • Rob
    Can you try the following please,

    On the machine running the agent, edit the jre/lib/healthcenter.properties file and change

    com.ibm.java.diagnostics.healthcenter.agent.transport=iiop to be
    com.ibm.java.diagnostics.healthcenter.agent.transport=jrmp

    Then restart the agent and see if that fixes the problem. We change the protocol for the network betweem 1.2 and 1.3 and I'm wondering if that has caused your problem.

    regards

    Toby
  • tomtommoiron
    tomtommoiron
    8 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-07-06T09:20:23Z  
    Can you try the following please,

    On the machine running the agent, edit the jre/lib/healthcenter.properties file and change

    com.ibm.java.diagnostics.healthcenter.agent.transport=iiop to be
    com.ibm.java.diagnostics.healthcenter.agent.transport=jrmp

    Then restart the agent and see if that fixes the problem. We change the protocol for the network betweem 1.2 and 1.3 and I'm wondering if that has caused your problem.

    regards

    Toby
    I try to change the agent transport on my installation. I use an open port between the agent host and the ISA host.
    I do not have anymore the orbtrc log file but the connection does not work. The ISA healthcenter module always display the No JVM detected on pn1inad02 on port 9084 aftern a timeout waiting message.

    It looks like the agent does not send the right response to healthcenter ISA module.
  • tomtommoiron
    tomtommoiron
    8 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-07-06T21:23:01Z  
    I try to change the agent transport on my installation. I use an open port between the agent host and the ISA host.
    I do not have anymore the orbtrc log file but the connection does not work. The ISA healthcenter module always display the No JVM detected on pn1inad02 on port 9084 aftern a timeout waiting message.

    It looks like the agent does not send the right response to healthcenter ISA module.
    To summarize:

    I do not use ssh tunneling anymore.
    I have changed iiop protocol with jrmp protocol in the healthcenter.properties file
    When I launch the jvm running the agent with -Xhealthcenter:port=9084 as command line option, I have a /tmp/healthcenter.31588536.log file and a /tmp/healthcenter.31588536.log.lck file
    When I stop the JVM, the lck file is deleted

    I cannot connect to the agent with the ISA.

    Any Idea?

    Is there a way to obtain more trace (on the agent or on the ISA or both) to find what is wrong in the communication between agent and ISA?

    Thanks for help
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-07-07T09:25:19Z  
    To summarize:

    I do not use ssh tunneling anymore.
    I have changed iiop protocol with jrmp protocol in the healthcenter.properties file
    When I launch the jvm running the agent with -Xhealthcenter:port=9084 as command line option, I have a /tmp/healthcenter.31588536.log file and a /tmp/healthcenter.31588536.log.lck file
    When I stop the JVM, the lck file is deleted

    I cannot connect to the agent with the ISA.

    Any Idea?

    Is there a way to obtain more trace (on the agent or on the ISA or both) to find what is wrong in the communication between agent and ISA?

    Thanks for help
    Hi,

    Can you try a simple test with the health center using the vm that is installed in WAS but not using WAS. So, from the AIX command line, run the following program and see what you get on the console

    ==============
    class HealthcenterTest{

    static public void main(String[] args) {
    try {
    System.out.println("press enter key to exit");
    System.in.read();
    } catch (Exception e) {
    }
    }
    }
    ==============

    compile and run it with

    java -Xhealthcenter HealthcenterTest

    You should see some info appear telling you the port it is running on. Then, without terminating the program, can you try and connect the gui to it through your network making sure the port is open on any firewalls you have

    regards

    Toby
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-09-27T10:35:39Z  
    Hi,

    Can you try a simple test with the health center using the vm that is installed in WAS but not using WAS. So, from the AIX command line, run the following program and see what you get on the console

    ==============
    class HealthcenterTest{

    static public void main(String[] args) {
    try {
    System.out.println("press enter key to exit");
    System.in.read();
    } catch (Exception e) {
    }
    }
    }
    ==============

    compile and run it with

    java -Xhealthcenter HealthcenterTest

    You should see some info appear telling you the port it is running on. Then, without terminating the program, can you try and connect the gui to it through your network making sure the port is open on any firewalls you have

    regards

    Toby
    I noticed the same problem with SSH tunneling and also when the port defined in HC agent was opened in the firewall and no tunneling was needed. I used Wireshark to capture data between the two machines and saw that the initialization on the HC session is built with that port, but then the actual data is transferred between two random(?) ports. So, the port defined in the HC agent isn't used and that's why the SSH tunneling doesn't work and neither does opening one port in the firewall.
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-10-20T09:01:14Z  
    I noticed the same problem with SSH tunneling and also when the port defined in HC agent was opened in the firewall and no tunneling was needed. I used Wireshark to capture data between the two machines and saw that the initialization on the HC session is built with that port, but then the actual data is transferred between two random(?) ports. So, the port defined in the HC agent isn't used and that's why the SSH tunneling doesn't work and neither does opening one port in the firewall.
    Changing com.ibm.java.diagnostics.healthcenter.agent.transport=iiop to
    com.ibm.java.diagnostics.healthcenter.agent.transport=jrmp solved the problem. In my case the file was located at /opt/IBM/WebSphere/AppServer/java/jre/lib/healthcenter.properties.
  • peacocb
    peacocb
    22 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-10-20T09:56:55Z  
    Changing com.ibm.java.diagnostics.healthcenter.agent.transport=iiop to
    com.ibm.java.diagnostics.healthcenter.agent.transport=jrmp solved the problem. In my case the file was located at /opt/IBM/WebSphere/AppServer/java/jre/lib/healthcenter.properties.
    Hi "niekko",

    Just to let you know what is happening "under the covers" ...

    When you select "iiop" this means the Orb is going to handle the traffic between the client and the server. Health Center uses the defined port to make the initial contact and then the Orb creates a new, "random" port to handle the data traffic.

    When you specify "jrmp" this means the Health Center client contacts the server on the defined port to make the initial contact and then the server uses this port for all the data traffic.

    This "random port or the Orb" is being looked at as it has caused a number of customers the same problem as you ... for which I apologise. I'm glad to hear you have got it working.

    Brian Peacock
  • lcarder
    lcarder
    1 Post

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2012-07-18T22:06:27Z  
    • peacocb
    • ‏2011-10-20T09:56:55Z
    Hi "niekko",

    Just to let you know what is happening "under the covers" ...

    When you select "iiop" this means the Orb is going to handle the traffic between the client and the server. Health Center uses the defined port to make the initial contact and then the Orb creates a new, "random" port to handle the data traffic.

    When you specify "jrmp" this means the Health Center client contacts the server on the defined port to make the initial contact and then the server uses this port for all the data traffic.

    This "random port or the Orb" is being looked at as it has caused a number of customers the same problem as you ... for which I apologise. I'm glad to hear you have got it working.

    Brian Peacock
    I am running into the same issue on a new installation despite changing the jrmp. I have to change the port assignment to a port open through my firewall, which appears to be working according to the log and netstat. When I attempt the connection, I see an established connection from my laptop to the agent port, but I am still returned the 'No JVM detected...' error after 60 seconds. My java version is as follows:

    java version "1.6.0"
    Java(TM) SE Runtime Environment (build pap6460sr10fp1-20120321_01(SR10 FP1))
    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)
    J9VM - 20120202_101568
    JIT - r9_20111107_21307ifx1
    GC - 20120202_AA)
    JCL - 20120320_01

    Thanks,
    Layne
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2012-07-19T07:13:59Z  
    • lcarder
    • ‏2012-07-18T22:06:27Z
    I am running into the same issue on a new installation despite changing the jrmp. I have to change the port assignment to a port open through my firewall, which appears to be working according to the log and netstat. When I attempt the connection, I see an established connection from my laptop to the agent port, but I am still returned the 'No JVM detected...' error after 60 seconds. My java version is as follows:

    java version "1.6.0"
    Java(TM) SE Runtime Environment (build pap6460sr10fp1-20120321_01(SR10 FP1))
    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)
    J9VM - 20120202_101568
    JIT - r9_20111107_21307ifx1
    GC - 20120202_AA)
    JCL - 20120320_01

    Thanks,
    Layne
    Hi,

    Can you confirm what level of the health center agent you are using ? You can find this out by typing java -Xhealthcenter -version at the command line.

    Is it possible to disable the firewall just for a test to see if is that that is blocking the communication ?

    regards

    Toby
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2012-12-07T12:48:02Z  
    Hi,

    Can you confirm what level of the health center agent you are using ? You can find this out by typing java -Xhealthcenter -version at the command line.

    Is it possible to disable the firewall just for a test to see if is that that is blocking the communication ?

    regards

    Toby
    Hi

    I founf a settings which works with tunneling and so firewall using iiop

    if server A is the server on which HealthCenter is installed (sorry my workstation is Linux, you will need to find equivalent commands if you use Windows)
    and server B is WebSphere Application server

    ON SERVER B :
    set these properties in WAS JVM parameters (and after installing HealthCenter agent) :
    -D com.ibm.java.diagnostics.healthcenter.agent.port=1972 -Dcom.ibm.java.diagnostics.healthcenter.agent.iiop.port=1975
    On SERVER A :
    Set an IP alias on server A as user root :
    /sbin/ifconfig eth0:0 <IP@ of server B>

    Commands to enable tunneling :
    ssh -4 -f user@<NAT IP@ of server B> -L <IP@ of server B>:1972:<IP@ of server B>:1972 -N
    ssh -4 -f user@<NAT IP@ of server B> -L <IP@ of server B>:1975:<IP@ of server B>:1975 -N

    you will be able to use HealthCenter specifying <IP@ of server B> as Hostname and 1972 as Port

    Regards,
    Jean-Pierre
  • ShahedM
    ShahedM
    3 Posts

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2014-07-18T11:10:05Z  
    • peacocb
    • ‏2011-10-20T09:56:55Z
    Hi "niekko",

    Just to let you know what is happening "under the covers" ...

    When you select "iiop" this means the Orb is going to handle the traffic between the client and the server. Health Center uses the defined port to make the initial contact and then the Orb creates a new, "random" port to handle the data traffic.

    When you specify "jrmp" this means the Health Center client contacts the server on the defined port to make the initial contact and then the server uses this port for all the data traffic.

    This "random port or the Orb" is being looked at as it has caused a number of customers the same problem as you ... for which I apologise. I'm glad to hear you have got it working.

    Brian Peacock

    Brian,

     

    Thanks for your detailed explanation,

    Its 2014 now and I am using WAS 8.5.5 and the very latest healthcenter bundled with ISA v5.

    I too ran into this issue and fixed it by using JRMP instead of IIOP.

    Is this problem still present with WAS (or has it been fixed and I am doing something incorrect) ? The docs dont seem to reflect this, if this is the case.

     

    Thanks

  • RoddiPotter
    RoddiPotter
    1 Post

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2014-10-31T17:45:40Z  
    • ShahedM
    • ‏2014-07-18T11:10:05Z

    Brian,

     

    Thanks for your detailed explanation,

    Its 2014 now and I am using WAS 8.5.5 and the very latest healthcenter bundled with ISA v5.

    I too ran into this issue and fixed it by using JRMP instead of IIOP.

    Is this problem still present with WAS (or has it been fixed and I am doing something incorrect) ? The docs dont seem to reflect this, if this is the case.

     

    Thanks

    JRMP works for me too using the latest Eclipse plugin:

    I set com.ibm.java.diagnostics.healthcenter.agent.transport=jrmp in healthcenter.properties.  I also added the -Xhealthcenter in Generic JVM arguments through the WAS Admin Console.  After server startup, I can connect from Eclipse on port 1972.

    I'm running WAS 8.5.5 on my local machine with Eclipse (Luna) Health Center plugin on Windows 7 64 bit.  Too bad the docs indicate JRMP is for z/OS as I ended up on a bit of a goose chase until I found ShahedM's post.  I tried many combinations of various settings using IIOP and nothing worked until I set the transport to JRMP.

    Thanks!

    Roddi