Topic
  • 10 replies
  • Latest Post - ‏2012-01-31T12:36:29Z by SystemAdmin
Rob_Nugent
Rob_Nugent
7 Posts

Pinned topic Problems with HealthCenter agent on AIX in headless mode.

‏2012-01-30T11:23:34Z |
Hi,

I've got some problems running the healthcenter agent in headless mode.

As background, we have a server application, running JVMs in many hundred processes, and I'd like to extract some profiling information. The customer's firewall is making attaching from the ISA problematic, hence I want to use headless mode.

The server processes are started with a common set of JVM options (we have no control on a per-process basis). java -Xhealthcenter -version shows the agent version as:

INFO: Agent version "2.0.0.20111124"

I'm starting up with the following options:

-Xhealthcenter
-Djava.rmi.server.hostname=129.153.201.82
-Dcom.ibm.java.diagnostics.healthcenter.data.collection.level=headless
-Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=1
-Dcom.ibm.java.diagnostics.healthcenter.headless.output.directory=/tmp/hc
Problem 1:
==========

I see files emitted into /tmp/hc, but then the .hcd files fail to get created. I see multiple exceptions like:

WARNING: Error writing to file {0}
Throwable occurred: java.io.FileNotFoundException: JLASource835818_1 (A file or directory in the path name does not exist.)
at java.io.FileInputStream.<init>(FileInputStream.java:123)
at java.io.FileInputStream.<init>(FileInputStream.java:83)
at com.ibm.java.diagnostics.healthcenter.agent.mbean.HeadlessAgent.createHCDFile(HeadlessAgent.java:271)
at com.ibm.java.diagnostics.healthcenter.agent.mbean.HeadlessAgent.access$100(HeadlessAgent.java:30)
at com.ibm.java.diagnostics.healthcenter.agent.mbean.HeadlessAgent$TimingControlThread.run(HeadlessAgent.java:483)
Problem 2:
==========

If I remove the parameter specifying the /tmp/hc directory, the .hcd files get created OK in the current directory (which is undesirable for me due to their number). However, if I take the 'hcd' files for a single process, while the first 'hcd' file for each process seems to contain some of the method names, the later files (which I am most interested in), lack the method names completely and are therefore useless. I am attaching one of these.

If anyone has any advise on any of this it would be very much appreciated.

Regards,
Rob
Updated on 2012-01-31T12:36:29Z at 2012-01-31T12:36:29Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-30T12:59:16Z  
    Hi Rob,

    In answer to your problems, does the directory /tmp/hc have full permissions to read/write to (I imagine it does, jsut thought I would check)? I'll investigate this further in the meantime.

    For the missing method names, this is unfortunately a limitation based on the level of the vm you are using. The problem is that the data that we use to cross reference a method name with its hex value is only produced once. So only the .hcd file containing that value will be able to do the mapping between name and hex value. In subsequent .hcd files, they will not contain that value so you only get hex values. There is a change in java 626 sr1 and Java 7 that allows the health center to get at that information when required but without the latest vm, that data is inavailable.

    A workaround that may help you is to amend the .hcd files that are created. These are basically just zip files. One of the file in that .hcd is called MethodDictionarySource<pid>. You could try replacing some of the those in the later .hcd files with the one from the first one as that will contain the data you are missing. You could even try just adding that file to the .hcd so that the .hcd you load will contain 2 MethodDictionarySource files.

    regards

    Toby
  • Rob_Nugent
    Rob_Nugent
    7 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-30T16:09:50Z  
    Hi Rob,

    In answer to your problems, does the directory /tmp/hc have full permissions to read/write to (I imagine it does, jsut thought I would check)? I'll investigate this further in the meantime.

    For the missing method names, this is unfortunately a limitation based on the level of the vm you are using. The problem is that the data that we use to cross reference a method name with its hex value is only produced once. So only the .hcd file containing that value will be able to do the mapping between name and hex value. In subsequent .hcd files, they will not contain that value so you only get hex values. There is a change in java 626 sr1 and Java 7 that allows the health center to get at that information when required but without the latest vm, that data is inavailable.

    A workaround that may help you is to amend the .hcd files that are created. These are basically just zip files. One of the file in that .hcd is called MethodDictionarySource<pid>. You could try replacing some of the those in the later .hcd files with the one from the first one as that will contain the data you are missing. You could even try just adding that file to the .hcd so that the .hcd you load will contain 2 MethodDictionarySource files.

    regards

    Toby
    Tony,

    Many thanks for the prompt response.

    1) the /tmp/hc directory has 755 permissons for the user running the JVM. Files get created in this directory, but the zipping up into the 'hcd' fails, leaving a zero-length file. i.e. it doesn't look like a permissions issue. e.g. I see the following after it has tried to emit the first hcd file for each process:

    $ ls -l /tmp/hc
    total 552
    -rw-r--r-- 1 rn133959 bteng 18900 Jan 30 04:06 EnvironmentSource802914_2
    -rw-r--r-- 1 rn133959 bteng 18900 Jan 30 04:06 EnvironmentSource835818_2
    -rw-r--r-- 1 rn133959 bteng 24220 Jan 30 04:06 JLASource802914_2
    -rw-r--r-- 1 rn133959 bteng 26141 Jan 30 04:06 JLASource835818_2
    -rw-r--r-- 1 rn133959 bteng 1305 Jan 30 04:07 MemorySource802914_2
    -rw-r--r-- 1 rn133959 bteng 1305 Jan 30 04:07 MemorySource835818_2
    -rw-r--r-- 1 rn133959 bteng 24 Jan 30 04:06 MethodDictionarySource802914_2
    -rw-r--r-- 1 rn133959 bteng 24 Jan 30 04:06 MethodDictionarySource835818_2
    -rw-r--r-- 1 rn133959 bteng 76416 Jan 30 04:06 TRACESubscriberSource802914_2
    -rw-r--r-- 1 rn133959 bteng 84620 Jan 30 04:06 TRACESubscriberSource835818_2
    -rw-r--r-- 1 rn133959 bteng 1483 Jan 30 04:07 ThreadsSource802914_2
    -rw-r--r-- 1 rn133959 bteng 1488 Jan 30 04:06 ThreadsSource835818_2
    -rw-r--r-- 1 rn133959 bteng 0 Jan 30 04:06 healthcenter802914_1.hcd
    -rw-r--r-- 1 rn133959 bteng 0 Jan 30 04:06 healthcenter835818_1.hcd

    2) I'm not sure what you mean by 'java 626 sr1'. Can you explain further please ? I'm on 'SR9' of Java 6:

    $ java -version
    java version "1.6.0"
    Java(TM) SE Runtime Environment (build pap3260sr9fp2-20110627_03(SR9 FP2))
    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc-32 jvmap3260sr9-20110624_85526 (JIT enabled, AOT enabled)
    J9VM - 20110624_085526
    JIT - r9_20101028_17488ifx17
    GC - 20101027_AA)
    JCL - 20110530_01

    It looks like SR10 is on the IBM website. Not sure how either of these correspond to the version you quote ?

    3) Thanks for the MethodDictionary suggestion. That seems to suggest that the hex method values should be consistent across all hcd files emitted for one process. I'm I understanding correctly ?

    Again many thanks for your time and effort in responding.
    Regards,
    Rob
  • Rob_Nugent
    Rob_Nugent
    7 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-30T16:14:09Z  
    Hi Rob,

    In answer to your problems, does the directory /tmp/hc have full permissions to read/write to (I imagine it does, jsut thought I would check)? I'll investigate this further in the meantime.

    For the missing method names, this is unfortunately a limitation based on the level of the vm you are using. The problem is that the data that we use to cross reference a method name with its hex value is only produced once. So only the .hcd file containing that value will be able to do the mapping between name and hex value. In subsequent .hcd files, they will not contain that value so you only get hex values. There is a change in java 626 sr1 and Java 7 that allows the health center to get at that information when required but without the latest vm, that data is inavailable.

    A workaround that may help you is to amend the .hcd files that are created. These are basically just zip files. One of the file in that .hcd is called MethodDictionarySource<pid>. You could try replacing some of the those in the later .hcd files with the one from the first one as that will contain the data you are missing. You could even try just adding that file to the .hcd so that the .hcd you load will contain 2 MethodDictionarySource files.

    regards

    Toby
    Tony,

    FYI - I am appending the _1 and _2 suffix 'hcd' files that match the _3 one I attached earlier in case they are of any use to you. N.B. this is from my test system, not the real customer system.

    Rob
  • Rob_Nugent
    Rob_Nugent
    7 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-30T16:14:32Z  
    Tony,

    FYI - I am appending the _1 and _2 suffix 'hcd' files that match the _3 one I attached earlier in case they are of any use to you. N.B. this is from my test system, not the real customer system.

    Rob
    And the _1
  • Rob_Nugent
    Rob_Nugent
    7 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-30T16:33:09Z  
    And the _1
    Tony,

    Just to confirm, the trick of replacing the MethodDictionarySource files in the 'hcd' (zip) doesn't seem to work.

    In each of the files the MethodDictionarySource file is just 24 bytes consisting of:

    #MethodDictionarySource

    Please see attched for details.

    Rob
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-31T08:33:13Z  
    Tony,

    Just to confirm, the trick of replacing the MethodDictionarySource files in the 'hcd' (zip) doesn't seem to work.

    In each of the files the MethodDictionarySource file is just 24 bytes consisting of:

    #MethodDictionarySource

    Please see attched for details.

    Rob
    I notice that you have the -Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=1 set rather than leaving it at the default of '0'. A '1' means you will get a set of files that each one will cover 1 minute. I would recommend leaving this at '0' so you get one set of files covering the whole run. This will give you a much better view of the profiling (and will help with the method names problem). Unless of course you specifically want 1 minute data collections.

    Java 626 sr1 refers to Java 6 running on a 2.6 vm, Java 6 sr9 runs on a 2.4 vm as does Java 6 sr 10. The vm's with the method profiling change in currently only ship within WebSphere. The next release of Java 7 will contain this change but that is also not available yet.

    regards

    Toby
  • Rob_Nugent
    Rob_Nugent
    7 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-31T10:22:46Z  
    I notice that you have the -Dcom.ibm.java.diagnostics.healthcenter.headless.run.duration=1 set rather than leaving it at the default of '0'. A '1' means you will get a set of files that each one will cover 1 minute. I would recommend leaving this at '0' so you get one set of files covering the whole run. This will give you a much better view of the profiling (and will help with the method names problem). Unless of course you specifically want 1 minute data collections.

    Java 626 sr1 refers to Java 6 running on a 2.6 vm, Java 6 sr9 runs on a 2.4 vm as does Java 6 sr 10. The vm's with the method profiling change in currently only ship within WebSphere. The next release of Java 7 will contain this change but that is also not available yet.

    regards

    Toby
    Tony,

    Thanks again for the reply.

    We're setting the 'run.duration' parameter due to a problem with the application server we are using - it fails to shut down the JVM at process end (it just calls 'exit()'), hence the 'hcd' files don't get created, although the 'intermediate' files do get emmited.

    Would it be safe to ZIP these files up into an 'hcd' ? I was worried that they are probably not getting closed properly when the process exits, so I wasn't sure of their integrity...

    Thanks,
    Rob
  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-31T11:12:06Z  
    Tony,

    Thanks again for the reply.

    We're setting the 'run.duration' parameter due to a problem with the application server we are using - it fails to shut down the JVM at process end (it just calls 'exit()'), hence the 'hcd' files don't get created, although the 'intermediate' files do get emmited.

    Would it be safe to ZIP these files up into an 'hcd' ? I was worried that they are probably not getting closed properly when the process exits, so I wasn't sure of their integrity...

    Thanks,
    Rob
    Hi Rob,

    It is safe to zip the files up into a .hcd file, that is all the final stage does.

    regards

    Toby
  • Rob_Nugent
    Rob_Nugent
    7 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-31T12:31:32Z  
    Hi Rob,

    It is safe to zip the files up into a .hcd file, that is all the final stage does.

    regards

    Toby
    Tony,

    Thanks - I just tried that experiment on my test system, and it seems to work, but I'm still not getting all the method names, despite the fact that I removed the 'runtime.duration' parameter. I thought/hoped this would/should give me all the method names. Did I misunderstand you previous comments, or are there still circumstances that I'd need the 2.6 JVM to see all these ?

    I'm attaching my manually zipped up 'b.hcd' so that you can see this, if you are interested...

    Thanks again,
    Rob

    Attachments

  • SystemAdmin
    SystemAdmin
    188 Posts

    Re: Problems with HealthCenter agent on AIX in headless mode.

    ‏2012-01-31T12:36:29Z  
    Tony,

    Thanks - I just tried that experiment on my test system, and it seems to work, but I'm still not getting all the method names, despite the fact that I removed the 'runtime.duration' parameter. I thought/hoped this would/should give me all the method names. Did I misunderstand you previous comments, or are there still circumstances that I'd need the 2.6 JVM to see all these ?

    I'm attaching my manually zipped up 'b.hcd' so that you can see this, if you are interested...

    Thanks again,
    Rob
    There will still be some missing method names but these should all be restricted to class library names (java.lang.String methods for example) as these may still be loaded by the vm before the health center agent starts. Application specific methods should all be present as these will be loaded after the health center agent is running.

    regards

    Toby