Topic
10 replies Latest Post - ‏2012-01-31T12:36:29Z by SystemAdmin
Rob_Nugent
Rob_Nugent
7 Posts
ACCEPTED ANSWER

Pinned topic Problems with HealthCenter agent on AIX in headless mode.

‏2012-01-30T11:23:34Z |
Hi,

I've got some problems running the healthcenter agent in headless mode.

As background, we have a server application, running JVMs in many hundred processes, and I'd like to extract some profiling information. The customer's firewall is making attaching from the ISA problematic, hence I want to use headless mode.

The server processes are started with a common set of JVM options (we have no control on a per-process basis). java -Xhealthcenter -version shows the agent version as:

INFO: Agent version "2.0.0.20111124"

I'm starting up with the following options:

-Xhealthcenter
-Djava.rmi.server.hostname=129.153.201.82
-Dcom.ibm.java.diagnostics.healthcenter.data.collection.level=headless
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=1
-Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=/tmp/hc
Problem 1:
==========

I see files emitted into /tmp/hc, but then the .hcd files fail to get created. I see multiple exceptions like:

WARNING: Error writing to file {0}
Throwable occurred: java.io.FileNotFoundException: JLASource835818_1 (A file or directory in the path name does not exist.)
at java.io.FileInputStream.<init>(FileInputStream.java:123)
at java.io.FileInputStream.<init>(FileInputStream.java:83)
at com.ibm.java.diagnostics.healthcenter.agent.mbean.HeadlessAgent.createHCDFile(HeadlessAgent.java:271)
at com.ibm.java.diagnostics.healthcenter.agent.mbean.HeadlessAgent.access$100(HeadlessAgent.java:30)
at com.ibm.java.diagnostics.healthcenter.agent.mbean.HeadlessAgent$TimingControlThread.run(HeadlessAgent.java:483)
Problem 2:
==========

If I remove the parameter specifying the /tmp/hc directory, the .hcd files get created OK in the current directory (which is undesirable for me due to their number). However, if I take the 'hcd' files for a single process, while the first 'hcd' file for each process seems to contain some of the method names, the later files (which I am most interested in), lack the method names completely and are therefore useless. I am attaching one of these.

If anyone has any advise on any of this it would be very much appreciated.

Regards,
Rob
Updated on 2012-01-31T12:36:29Z at 2012-01-31T12:36:29Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    188 Posts
    ACCEPTED ANSWER

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-30T12:59:16Z  in response to Rob_Nugent
    Hi Rob,

    In answer to your problems, does the directory /tmp/hc have full permissions to read/write to (I imagine it does, jsut thought I would check)? I'll investigate this further in the meantime.

    For the missing method names, this is unfortunately a limitation based on the level of the vm you are using. The problem is that the data that we use to cross reference a method name with its hex value is only produced once. So only the .hcd file containing that value will be able to do the mapping between name and hex value. In subsequent .hcd files, they will not contain that value so you only get hex values. There is a change in java 626 sr1 and Java 7 that allows the health center to get at that information when required but without the latest vm, that data is inavailable.

    A workaround that may help you is to amend the .hcd files that are created. These are basically just zip files. One of the file in that .hcd is called MethodDictionarySource<pid>. You could try replacing some of the those in the later .hcd files with the one from the first one as that will contain the data you are missing. You could even try just adding that file to the .hcd so that the .hcd you load will contain 2 MethodDictionarySource files.

    regards

    Toby
    • Rob_Nugent
      Rob_Nugent
      7 Posts
      ACCEPTED ANSWER

      Re: Problems with HealthCenter agent on AIX in headless mode.

      ‏2012-01-30T16:09:50Z  in response to SystemAdmin
      Tony,

      Many thanks for the prompt response.

      1) the /tmp/hc directory has 755 permissons for the user running the JVM. Files get created in this directory, but the zipping up into the 'hcd' fails, leaving a zero-length file. i.e. it doesn't look like a permissions issue. e.g. I see the following after it has tried to emit the first hcd file for each process:

      $ ls -l /tmp/hc
      total 552
      -rw-r--r-- 1 rn133959 bteng 18900 Jan 30 04:06 EnvironmentSource802914_2
      -rw-r--r-- 1 rn133959 bteng 18900 Jan 30 04:06 EnvironmentSource835818_2
      -rw-r--r-- 1 rn133959 bteng 24220 Jan 30 04:06 JLASource802914_2
      -rw-r--r-- 1 rn133959 bteng 26141 Jan 30 04:06 JLASource835818_2
      -rw-r--r-- 1 rn133959 bteng 1305 Jan 30 04:07 MemorySource802914_2
      -rw-r--r-- 1 rn133959 bteng 1305 Jan 30 04:07 MemorySource835818_2
      -rw-r--r-- 1 rn133959 bteng 24 Jan 30 04:06 MethodDictionarySource802914_2
      -rw-r--r-- 1 rn133959 bteng 24 Jan 30 04:06 MethodDictionarySource835818_2
      -rw-r--r-- 1 rn133959 bteng 76416 Jan 30 04:06 TRACESubscriberSource802914_2
      -rw-r--r-- 1 rn133959 bteng 84620 Jan 30 04:06 TRACESubscriberSource835818_2
      -rw-r--r-- 1 rn133959 bteng 1483 Jan 30 04:07 ThreadsSource802914_2
      -rw-r--r-- 1 rn133959 bteng 1488 Jan 30 04:06 ThreadsSource835818_2
      -rw-r--r-- 1 rn133959 bteng 0 Jan 30 04:06 healthcenter802914_1.hcd
      -rw-r--r-- 1 rn133959 bteng 0 Jan 30 04:06 healthcenter835818_1.hcd

      2) I'm not sure what you mean by 'java 626 sr1'. Can you explain further please ? I'm on 'SR9' of Java 6:

      $ java -version
      java version "1.6.0"
      Java(TM) SE Runtime Environment (build pap3260sr9fp2-20110627_03(SR9 FP2))
      IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc-32 jvmap3260sr9-20110624_85526 (JIT enabled, AOT enabled)
      J9VM - 20110624_085526
      JIT - r9_20101028_17488ifx17
      GC - 20101027_AA)
      JCL - 20110530_01

      It looks like SR10 is on the IBM website. Not sure how either of these correspond to the version you quote ?

      3) Thanks for the MethodDictionary suggestion. That seems to suggest that the hex method values should be consistent across all hcd files emitted for one process. I'm I understanding correctly ?

      Again many thanks for your time and effort in responding.
      Regards,
      Rob
    • Rob_Nugent
      Rob_Nugent
      7 Posts
      ACCEPTED ANSWER

      Re: Problems with HealthCenter agent on AIX in headless mode.

      ‏2012-01-30T16:14:09Z  in response to SystemAdmin
      Tony,

      FYI - I am appending the _1 and _2 suffix 'hcd' files that match the _3 one I attached earlier in case they are of any use to you. N.B. this is from my test system, not the real customer system.

      Rob
      • Rob_Nugent
        Rob_Nugent
        7 Posts
        ACCEPTED ANSWER

        Re: Problems with HealthCenter agent on AIX in headless mode.

        ‏2012-01-30T16:14:32Z  in response to Rob_Nugent
        And the _1
        • Rob_Nugent
          Rob_Nugent
          7 Posts
          ACCEPTED ANSWER

          Re: Problems with HealthCenter agent on AIX in headless mode.

          ‏2012-01-30T16:33:09Z  in response to Rob_Nugent
          Tony,

          Just to confirm, the trick of replacing the MethodDictionarySource files in the 'hcd' (zip) doesn't seem to work.

          In each of the files the MethodDictionarySource file is just 24 bytes consisting of:

          #MethodDictionarySource

          Please see attched for details.

          Rob
          • SystemAdmin
            SystemAdmin
            188 Posts
            ACCEPTED ANSWER

            Re: Problems with HealthCenter agent on AIX in headless mode.

            ‏2012-01-31T08:33:13Z  in response to Rob_Nugent
            I notice that you have the -Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=1 set rather than leaving it at the default of '0'. A '1' means you will get a set of files that each one will cover 1 minute. I would recommend leaving this at '0' so you get one set of files covering the whole run. This will give you a much better view of the profiling (and will help with the method names problem). Unless of course you specifically want 1 minute data collections.

            Java 626 sr1 refers to Java 6 running on a 2.6 vm, Java 6 sr9 runs on a 2.4 vm as does Java 6 sr 10. The vm's with the method profiling change in currently only ship within WebSphere. The next release of Java 7 will contain this change but that is also not available yet.

            regards

            Toby
            • Rob_Nugent
              Rob_Nugent
              7 Posts
              ACCEPTED ANSWER

              Re: Problems with HealthCenter agent on AIX in headless mode.

              ‏2012-01-31T10:22:46Z  in response to SystemAdmin
              Tony,

              Thanks again for the reply.

              We're setting the 'run.duration' parameter due to a problem with the application server we are using - it fails to shut down the JVM at process end (it just calls 'exit()'), hence the 'hcd' files don't get created, although the 'intermediate' files do get emmited.

              Would it be safe to ZIP these files up into an 'hcd' ? I was worried that they are probably not getting closed properly when the process exits, so I wasn't sure of their integrity...

              Thanks,
              Rob
              • SystemAdmin
                SystemAdmin
                188 Posts
                ACCEPTED ANSWER

                Re: Problems with HealthCenter agent on AIX in headless mode.

                ‏2012-01-31T11:12:06Z  in response to Rob_Nugent
                Hi Rob,

                It is safe to zip the files up into a .hcd file, that is all the final stage does.

                regards

                Toby
                • Rob_Nugent
                  Rob_Nugent
                  7 Posts
                  ACCEPTED ANSWER

                  Re: Problems with HealthCenter agent on AIX in headless mode.

                  ‏2012-01-31T12:31:32Z  in response to SystemAdmin
                  Tony,

                  Thanks - I just tried that experiment on my test system, and it seems to work, but I'm still not getting all the method names, despite the fact that I removed the 'runtime.duration' parameter. I thought/hoped this would/should give me all the method names. Did I misunderstand you previous comments, or are there still circumstances that I'd need the 2.6 JVM to see all these ?

                  I'm attaching my manually zipped up 'b.hcd' so that you can see this, if you are interested...

                  Thanks again,
                  Rob

                  Attachments

                  • SystemAdmin
                    SystemAdmin
                    188 Posts
                    ACCEPTED ANSWER

                    Re: Problems with HealthCenter agent on AIX in headless mode.

                    ‏2012-01-31T12:36:29Z  in response to Rob_Nugent
                    There will still be some missing method names but these should all be restricted to class library names (java.lang.String methods for example) as these may still be loaded by the vm before the health center agent starts. Application specific methods should all be present as these will be loaded after the health center agent is running.

                    regards

                    Toby