Topic
17 replies Latest Post - ‏2012-12-07T12:48:02Z by SystemAdmin
tomtommoiron
tomtommoiron
8 Posts
ACCEPTED ANSWER

Pinned topic Health Center. No JVM detected on localhost on port 1972

‏2011-06-30T09:37:27Z |
I installed the Agent on a AIX WebSphere and the JVM start seems to be correct. It listens on the 1972 port. Validated with netstat.
I installed the ISA HealthCenter tool. I verify with netstat that the connection is possible between the 2 hosts.

But I have a No JVM detected on localhost on port 1972 message on the ISA when I try to connect.

Here an extract of the netstats I perform while trying to connect
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 7 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 7 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 7 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 16 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 68 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 68 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 68 0 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 68 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED
tcp4 0 324 127.0.0.1.1972 127.0.0.1.57087 ESTABLISHED
tcp4 0 0 127.0.0.1.57087 127.0.0.1.1972 ESTABLISHED

And what I see in the /tmp/healthcenter.*.log
Jun 30, 2011 10:18:44 AM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgent
INFO: Health Center agent started on port 1972.
Jun 30, 2011 10:18:44 AM com.ibm.java.diagnostics.healthcenter.agent.mbean.HCLaunchMBean startAgent
INFO: Agent version "1.3.0.20110107"

I also have an empty /tmp/healthcenter.*.log.lck

Is there a way to add health center logs to ind what is wrong? In which log file should I found more detailed logs?

Thanks for help
Updated on 2012-12-07T12:48:02Z at 2012-12-07T12:48:02Z by SystemAdmin
  • tomtommoiron
    tomtommoiron
    8 Posts
    ACCEPTED ANSWER

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T09:39:49Z  in response to tomtommoiron
    Is there a way to add health center logs to find what is the cause of the connection issue? In which log file should I found more detailed logs?
  • tomtommoiron
    tomtommoiron
    8 Posts
    ACCEPTED ANSWER

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T09:40:41Z  in response to tomtommoiron
    _
  • SystemAdmin
    SystemAdmin
    188 Posts
    ACCEPTED ANSWER

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-06-30T09:40:51Z  in response to tomtommoiron
    Hi,

    Are you using any sort of port tunnelling to get to the machine what is running the health center agent ?

    regards

    Toby
    • tomtommoiron
      tomtommoiron
      8 Posts
      ACCEPTED ANSWER

      Re: Health Center. No JVM detected on localhost on port 1972

      ‏2011-06-30T09:42:23Z  in response to SystemAdmin
      Yes, I use a ssh tunnel to connect from my ISA (on a windows system) to the WAS JVM (on a AIX System).
      • tomtommoiron
        tomtommoiron
        8 Posts
        ACCEPTED ANSWER

        Re: Health Center. No JVM detected on localhost on port 1972

        ‏2011-06-30T10:09:48Z  in response to tomtommoiron
        I notice a difference between connection attempt when the tunnel is on and when it is off.

        When the tunnel is on, I see a message Searching localhost on port 1972... timeout in xxx seconds and after, I have the No JVM detected on localhost on port 1972 message.
        When the tunnel is off, I directly see the No JVM detected on localhost on port 1972

        The tunnel is from the localhost port 1972 to the AIX host port 1972.
      • tomtommoiron
        tomtommoiron
        8 Posts
        ACCEPTED ANSWER

        Re: Health Center. No JVM detected on localhost on port 1972

        ‏2011-06-30T10:13:42Z  in response to tomtommoiron
        I notice a difference between connection attempt when the tunnel is on and when it is off.

        When the tunnel is on, I see a message Searching localhost on port 1972... timeout in xxx seconds and after, I have the No JVM detected on localhost on port 1972 message.
        When the tunnel is off, I directly see the No JVM detected on localhost on port 1972

        The tunnel is from the localhost port 1972 to the AIX host port 1972.
  • rrosseter
    rrosseter
    2 Posts
    ACCEPTED ANSWER

    Re: Health Center. No JVM detected on localhost on port 1972

    ‏2011-07-05T18:59:30Z  in response to tomtommoiron
    I'm having a very similar issue after trying to upgrade from v1.2 to v1.3.

    High level sequence of events.
    • Installed v1.2 late last year which has been working fine until now.
    • Updated client side to HC Agent release v1.3.20110506
    • Connected to JVMs without issue but recieved notice that the agent was out of date (which makes sense as it was the v1.2 HC agent).
    • Used Support Assistant Help module to download latest HC Agent but the files are only dated 1/7/2011.
    • Installed HC Agent by taking backups of all files contained within the agent download and unzipping the zipfile at the correct folder level.
    • Verified that all backed up libraries were present and that new HC agent files were also in place with 01/07/2011 date.
    • Restarted JVMs
    • Can no longer connect to the JVMs using the HC client.
    • Logs in /tmp also show that the HC agents are active (similar messages as tomtommoiron reported)
    • Telnet tests to corresponding ports whre HC agents are supposed to be active are successful from my desktop.

    Configuration: P570 micropartition frames with 48GB memory running on 64bit AIX 5.3.11.5 TL11 with WAS 32bit http://ND 6.1.0.35 cf351044.07 vertically/horizontally clustered across 2 application servers running 4 JVMs each.

    • Rob
    • SystemAdmin
      SystemAdmin
      188 Posts
      ACCEPTED ANSWER

      Re: Health Center. No JVM detected on localhost on port 1972

      ‏2011-07-06T08:08:02Z  in response to rrosseter
      Can you try the following please,

      On the machine running the agent, edit the jre/lib/healthcenter.properties file and change

      com.ibm.java.diagnostics.healthcenter.agent.transport=iiop to be
      com.ibm.java.diagnostics.healthcenter.agent.transport=jrmp

      Then restart the agent and see if that fixes the problem. We change the protocol for the network betweem 1.2 and 1.3 and I'm wondering if that has caused your problem.

      regards

      Toby
      • tomtommoiron
        tomtommoiron
        8 Posts
        ACCEPTED ANSWER

        Re: Health Center. No JVM detected on localhost on port 1972

        ‏2011-07-06T09:20:23Z  in response to SystemAdmin
        I try to change the agent transport on my installation. I use an open port between the agent host and the ISA host.
        I do not have anymore the orbtrc log file but the connection does not work. The ISA healthcenter module always display the No JVM detected on pn1inad02 on port 9084 aftern a timeout waiting message.

        It looks like the agent does not send the right response to healthcenter ISA module.
        • tomtommoiron
          tomtommoiron
          8 Posts
          ACCEPTED ANSWER

          Re: Health Center. No JVM detected on localhost on port 1972

          ‏2011-07-06T21:23:01Z  in response to tomtommoiron
          To summarize:

          I do not use ssh tunneling anymore.
          I have changed iiop protocol with jrmp protocol in the healthcenter.properties file
          When I launch the jvm running the agent with -Xhealthcenter:port=9084 as command line option, I have a /tmp/healthcenter.31588536.log file and a /tmp/healthcenter.31588536.log.lck file
          When I stop the JVM, the lck file is deleted

          I cannot connect to the agent with the ISA.

          Any Idea?

          Is there a way to obtain more trace (on the agent or on the ISA or both) to find what is wrong in the communication between agent and ISA?

          Thanks for help
          • SystemAdmin
            SystemAdmin
            188 Posts
            ACCEPTED ANSWER

            Re: Health Center. No JVM detected on localhost on port 1972

            ‏2011-07-07T09:25:19Z  in response to tomtommoiron
            Hi,

            Can you try a simple test with the health center using the vm that is installed in WAS but not using WAS. So, from the AIX command line, run the following program and see what you get on the console

            ==============
            class HealthcenterTest{

            static public void main(String[] args) {
            try {
            System.out.println("press enter key to exit");
            System.in.read();
            } catch (Exception e) {
            }
            }
            }
            ==============

            compile and run it with

            java -Xhealthcenter HealthcenterTest

            You should see some info appear telling you the port it is running on. Then, without terminating the program, can you try and connect the gui to it through your network making sure the port is open on any firewalls you have

            regards

            Toby
            • SystemAdmin
              SystemAdmin
              188 Posts
              ACCEPTED ANSWER

              Re: Health Center. No JVM detected on localhost on port 1972

              ‏2011-09-27T10:35:39Z  in response to SystemAdmin
              I noticed the same problem with SSH tunneling and also when the port defined in HC agent was opened in the firewall and no tunneling was needed. I used Wireshark to capture data between the two machines and saw that the initialization on the HC session is built with that port, but then the actual data is transferred between two random(?) ports. So, the port defined in the HC agent isn't used and that's why the SSH tunneling doesn't work and neither does opening one port in the firewall.
              • SystemAdmin
                SystemAdmin
                188 Posts
                ACCEPTED ANSWER

                Re: Health Center. No JVM detected on localhost on port 1972

                ‏2011-10-20T09:01:14Z  in response to SystemAdmin
                Changing com.ibm.java.diagnostics.healthcenter.agent.transport=iiop to
                com.ibm.java.diagnostics.healthcenter.agent.transport=jrmp solved the problem. In my case the file was located at /opt/IBM/WebSphere/AppServer/java/jre/lib/healthcenter.properties.
                • peacocb
                  peacocb
                  22 Posts
                  ACCEPTED ANSWER

                  Re: Health Center. No JVM detected on localhost on port 1972

                  ‏2011-10-20T09:56:55Z  in response to SystemAdmin
                  Hi "niekko",

                  Just to let you know what is happening "under the covers" ...

                  When you select "iiop" this means the Orb is going to handle the traffic between the client and the server. Health Center uses the defined port to make the initial contact and then the Orb creates a new, "random" port to handle the data traffic.

                  When you specify "jrmp" this means the Health Center client contacts the server on the defined port to make the initial contact and then the server uses this port for all the data traffic.

                  This "random port or the Orb" is being looked at as it has caused a number of customers the same problem as you ... for which I apologise. I'm glad to hear you have got it working.

                  Brian Peacock
                  • lcarder
                    lcarder
                    1 Post
                    ACCEPTED ANSWER

                    Re: Health Center. No JVM detected on localhost on port 1972

                    ‏2012-07-18T22:06:27Z  in response to peacocb
                    I am running into the same issue on a new installation despite changing the jrmp. I have to change the port assignment to a port open through my firewall, which appears to be working according to the log and netstat. When I attempt the connection, I see an established connection from my laptop to the agent port, but I am still returned the 'No JVM detected...' error after 60 seconds. My java version is as follows:

                    java version "1.6.0"
                    Java(TM) SE Runtime Environment (build pap6460sr10fp1-20120321_01(SR10 FP1))
                    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr10fp1-20120202_101568 (JIT enabled, AOT enabled)
                    J9VM - 20120202_101568
                    JIT - r9_20111107_21307ifx1
                    GC - 20120202_AA)
                    JCL - 20120320_01

                    Thanks,
                    Layne
                    • SystemAdmin
                      SystemAdmin
                      188 Posts
                      ACCEPTED ANSWER

                      Re: Health Center. No JVM detected on localhost on port 1972

                      ‏2012-07-19T07:13:59Z  in response to lcarder
                      Hi,

                      Can you confirm what level of the health center agent you are using ? You can find this out by typing java -Xhealthcenter -version at the command line.

                      Is it possible to disable the firewall just for a test to see if is that that is blocking the communication ?

                      regards

                      Toby
                      • SystemAdmin
                        SystemAdmin
                        188 Posts
                        ACCEPTED ANSWER

                        Re: Health Center. No JVM detected on localhost on port 1972

                        ‏2012-12-07T12:48:02Z  in response to SystemAdmin
                        Hi

                        I founf a settings which works with tunneling and so firewall using iiop

                        if server A is the server on which HealthCenter is installed (sorry my workstation is Linux, you will need to find equivalent commands if you use Windows)
                        and server B is WebSphere Application server

                        ON SERVER B :
                        set these properties in WAS JVM parameters (and after installing HealthCenter agent) :
                        -D com.ibm.java.diagnostics.healthcenter.agent.port=1972 -Dcom.ibm.java.diagnostics.healthcenter.agent.iiop.port=1975
                        On SERVER A :
                        Set an IP alias on server A as user root :
                        /sbin/ifconfig eth0:0 <IP@ of server B>

                        Commands to enable tunneling :
                        ssh -4 -f user@<NAT IP@ of server B> -L <IP@ of server B>:1972:<IP@ of server B>:1972 -N
                        ssh -4 -f user@<NAT IP@ of server B> -L <IP@ of server B>:1975:<IP@ of server B>:1975 -N

                        you will be able to use HealthCenter specifying <IP@ of server B> as Hostname and 1972 as Port

                        Regards,
                        Jean-Pierre