Topic
  • 14 replies
  • Latest Post - ‏2013-07-11T14:13:28Z by DavidHarvey
DavidHarvey
DavidHarvey
10 Posts

Pinned topic File not found when starting JVM Server

‏2013-07-04T14:49:16Z |

I get the following error whenever I try to start a JVM in the CICS TS 5.1 Developer Trial version.

2013/07/04 07:58:48.438702 GMT *Exc*: Attempt to open JVM profile DFHOSGI has failed. Runtime error message = EDC5129I No such file
or directory.

 

The JVMPROFILEDIR=points to the directory containing the JVM Profile and USSHOME is also correct. This happens no matter what profile I try to use. The JVMtrace, JVMin and JVMout are all empty only out put is that from the dfhaxro module in jvmerr:

  

Storage Report for Enclave CELQPIPI Thu Jul  4 08:13:34 2013
Language Environment V01 R13.00

STACK64 statistics:
Initial size:                                               200M
Increment size:                                             100M
Maximum used by all concurrent threads:                     222M
Largest used by any thread:                                 222M
Number of increments allocated:                               1
64bit User HEAP statistics:
Initial size:                                               200M
Increment size:                                               4M

 

Not sure what is wrong with my set-up...

 

.

  • IvanHargreaves
    IvanHargreaves
    27 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-05T08:48:38Z  

    Hi David,

    The 8 char JVMPROFILE on your JVMSERVER resource must match exactly a file found in the JVMPROFILEDIR (specified in your SIT). It is a unix file, so it is case-sensitive. In CICS TS v5.1 the file must not have any extension. If you look in the JVMPROFILEDIR for that region and see a file called "DFHOSGI" then please check that the CICS userid has authority to access the file. If your CICS region id has access to read that file (you can use ls -al from a USS shell to check) then there must be something more fundamental.

    If you continue to have problems, set SJ and AP domain trace to 'ALL', and turn on auxtrace before enabling the JVMSERVER. If you can then get the cics auxtrace, the CICS job output, along with dfhjvmerr, dfhjvmout, and dfhjvmtrc if they exist, to me. I'll take a look.

    thx

    Ivan

  • DavidHarvey
    DavidHarvey
    10 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-05T11:28:13Z  

    Hi David,

    The 8 char JVMPROFILE on your JVMSERVER resource must match exactly a file found in the JVMPROFILEDIR (specified in your SIT). It is a unix file, so it is case-sensitive. In CICS TS v5.1 the file must not have any extension. If you look in the JVMPROFILEDIR for that region and see a file called "DFHOSGI" then please check that the CICS userid has authority to access the file. If your CICS region id has access to read that file (you can use ls -al from a USS shell to check) then there must be something more fundamental.

    If you continue to have problems, set SJ and AP domain trace to 'ALL', and turn on auxtrace before enabling the JVMSERVER. If you can then get the cics auxtrace, the CICS job output, along with dfhjvmerr, dfhjvmout, and dfhjvmtrc if they exist, to me. I'll take a look.

    thx

    Ivan

    Ivan - thanks for looking at this - I've attached some doc for you - I was using my own JVMProfiles Directory but find that even the supplied directory has this issue. The dfhjvmerr,dfhjvmout and dfhjvmtrc exist in the /tmp/ directory but are all empty. The attached trace is from the third attemt to enable the jvm in this run of CICS - so this time stamp:

     

    2013/07/05 10:29:05.613078 GMT *Exc*: Attempt to open JVM profile DFHOSGI has failed. Runtime error message = EDC5129I No such file
    or directory.

    Alos go this output one time when using the spplied JVMProfile directoy:

    2013/07/05 10:29:05.575583 GMT Entry: read_profile JVM Profile = 0/DFHOSGI
    2013/07/05 10:29:05.627150 GMT Event: Changing to = tmp

     

    Attachments

  • IvanHargreaves
    IvanHargreaves
    27 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-05T15:33:33Z  

    Ivan - thanks for looking at this - I've attached some doc for you - I was using my own JVMProfiles Directory but find that even the supplied directory has this issue. The dfhjvmerr,dfhjvmout and dfhjvmtrc exist in the /tmp/ directory but are all empty. The attached trace is from the third attemt to enable the jvm in this run of CICS - so this time stamp:

     

    2013/07/05 10:29:05.613078 GMT *Exc*: Attempt to open JVM profile DFHOSGI has failed. Runtime error message = EDC5129I No such file
    or directory.

    Alos go this output one time when using the spplied JVMProfile directoy:

    2013/07/05 10:29:05.575583 GMT Entry: read_profile JVM Profile = 0/DFHOSGI
    2013/07/05 10:29:05.627150 GMT Event: Changing to = tmp

     

    The message 0/DFHOSGI indicates a problem with the JVMPROFILEDIR. I notice you are pointing at the JVMProfiles directory in the CICS install tree. I would expect that location to be read-only? (which would explain the problem). However, if you are sure your CICS installation directories are no longer read-only then I will have to take a deeper look at the trace files provided.

    thx

    Ivan

  • DavidHarvey
    DavidHarvey
    10 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T10:01:52Z  

    The message 0/DFHOSGI indicates a problem with the JVMPROFILEDIR. I notice you are pointing at the JVMProfiles directory in the CICS install tree. I would expect that location to be read-only? (which would explain the problem). However, if you are sure your CICS installation directories are no longer read-only then I will have to take a deeper look at the trace files provided.

    thx

    Ivan

    Hi Ivan, I have run the test again with a different ZFS directory where the JVMProfiles are stored. I'm attaching 1) the CICS log for THCICT13, 2) the AUXTRACE and 3) some info regarding the ZFS permissions etc. Please let me know if you need anything else - thanks

  • DavidHarvey
    DavidHarvey
    10 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T10:27:04Z  

    Hi Ivan, I have run the test again with a different ZFS directory where the JVMProfiles are stored. I'm attaching 1) the CICS log for THCICT13, 2) the AUXTRACE and 3) some info regarding the ZFS permissions etc. Please let me know if you need anything else - thanks

     

    Attachments

  • DavidRoberts
    DavidRoberts
    2 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T12:07:12Z  

    Hi Ivan, I have run the test again with a different ZFS directory where the JVMProfiles are stored. I'm attaching 1) the CICS log for THCICT13, 2) the AUXTRACE and 3) some info regarding the ZFS permissions etc. Please let me know if you need anything else - thanks

    Hi David,

    With SJ and AP domain trace set to 'ALL' we would expect to see the trace from the enable of the JVM server in a file in the CICS job log.  The output is written to stdout so will be in the SYSPRINT DD if specified, or a SYSnnnnn generated DD.  The first line in the file should be something like:

    2013/07/09 11:52:48.039432 UTC DFHOSGI Entry: read_profile() - JVM Profile  = /u/cicsuser/JVMProfiles/DFHOSGI
    

    The trace we are looking for is in this file, not the aux trace.  Please check your trace settings again, enable the JVM server and check the CICS job log for the trace output.

    Thanks

    Dave

  • DavidHarvey
    DavidHarvey
    10 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T12:18:08Z  

    Hi David,

    With SJ and AP domain trace set to 'ALL' we would expect to see the trace from the enable of the JVM server in a file in the CICS job log.  The output is written to stdout so will be in the SYSPRINT DD if specified, or a SYSnnnnn generated DD.  The first line in the file should be something like:

    <pre dir="ltr">2013/07/09 11:52:48.039432 UTC DFHOSGI Entry: read_profile() - JVM Profile = /u/cicsuser/JVMProfiles/DFHOSGI </pre>

    The trace we are looking for is in this file, not the aux trace.  Please check your trace settings again, enable the JVM server and check the CICS job log for the trace output.

    Thanks

    Dave

    Hi Dave - I cut and pasted these into the attachment 'ZFS Permissions and USERIDs etc.docx' above SYS00002 ans SYS0003 appeared when attempting to enable the JVM - these also appear in the job log for THCICT13(THCICT13.TXT -above) - as shown here:

      DFHFC0201 07/09/2013 10:43:34 AE3BCITH Non-RLS file DFHCSD has been deallocated. Module DFHFCN.
     2013/07/09 09:32:55.104637 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file
     or directory.
     2013/07/09 09:36:32.616090 GMT Entry: read_profile JVM Profile = /DFHTSGI
     2013/07/09 09:36:32.653832 GMT Event: Changing to = tmp
     2013/07/09 09:36:32.641004 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file
     or directory.

     

    Let me know if you need anything else - thanks

  • DavidRoberts
    DavidRoberts
    2 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T12:48:37Z  

    Hi Dave - I cut and pasted these into the attachment 'ZFS Permissions and USERIDs etc.docx' above SYS00002 ans SYS0003 appeared when attempting to enable the JVM - these also appear in the job log for THCICT13(THCICT13.TXT -above) - as shown here:

      DFHFC0201 07/09/2013 10:43:34 AE3BCITH Non-RLS file DFHCSD has been deallocated. Module DFHFCN.
     2013/07/09 09:32:55.104637 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file
     or directory.
     2013/07/09 09:36:32.616090 GMT Entry: read_profile JVM Profile = /DFHTSGI
     2013/07/09 09:36:32.653832 GMT Event: Changing to = tmp
     2013/07/09 09:36:32.641004 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file
     or directory.

     

    Let me know if you need anything else - thanks

    OK, thanks.

    So, the strange thing is why your log shows /DFHTSGI rather than /TMVS/u/tstcict/JVMProfiles/DFHTSGI.  The message is created by concatenating JVMPROFILEDIR "/" JVMPROFILE.  So the JVMPROFILEDIR has somehow gone missing.

    So, to try and track down what's happening, please CEMT INQUIRE your JVM server.  The profile directory is shown in the expanded view.  If this looks OK, we probably need a dump to investigate further

    Thanks,

    Dave

     

  • DavidHarvey
    DavidHarvey
    10 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T13:01:47Z  

    OK, thanks.

    So, the strange thing is why your log shows /DFHTSGI rather than /TMVS/u/tstcict/JVMProfiles/DFHTSGI.  The message is created by concatenating JVMPROFILEDIR "/" JVMPROFILE.  So the JVMPROFILEDIR has somehow gone missing.

    So, to try and track down what's happening, please CEMT INQUIRE your JVM server.  The profile directory is shown in the expanded view.  If this looks OK, we probably need a dump to investigate further

    Thanks,

    Dave

     

    Dave,

    The expaned Cemt I JVM shows this :

    I JVM
    RESULT - OVERTYPE TO MODIFY
    Jvmserver(DFH$JVMS)
    Enablestatus( Disabled )
    Purgetype(              )
    Prfile(DFHTSGI)
    Lerunopts(DFHAXRO)
    Threadcount(000)
    Threadlimit( 015 )
    Currentheap(0)
    Initheap(0)
    Maxheap(0)
    Gcpolicy()
    Occupancy(0)
    Pid(0000000000)
    Profiledir(/TMVS/u/tstcict/JVMProfiles/)
    Installtime(07/09/13 13:43:16)
    Installusrid(HTS0201)
    Installagent(Csdapi)
    Definesource(THJVM)

    I JVM
    RESULT - OVERTYPE TO MODIFY
    Jvmserver(DFH$JVMS)
    Definesource(THJVM)
    Definetime(07/08/13 11:39:44)
    Changetime(07/09/13 10:32:42)
    Changeusrid(HTS0201)
    Changeagent(Csdapi)
    Changeagrel(0680)

    That being the case how should I capture the dump ? - thanks

  • IvanHargreaves
    IvanHargreaves
    27 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T17:08:37Z  

    Dave,

    The expaned Cemt I JVM shows this :

    I JVM
    RESULT - OVERTYPE TO MODIFY
    Jvmserver(DFH$JVMS)
    Enablestatus( Disabled )
    Purgetype(              )
    Prfile(DFHTSGI)
    Lerunopts(DFHAXRO)
    Threadcount(000)
    Threadlimit( 015 )
    Currentheap(0)
    Initheap(0)
    Maxheap(0)
    Gcpolicy()
    Occupancy(0)
    Pid(0000000000)
    Profiledir(/TMVS/u/tstcict/JVMProfiles/)
    Installtime(07/09/13 13:43:16)
    Installusrid(HTS0201)
    Installagent(Csdapi)
    Definesource(THJVM)

    I JVM
    RESULT - OVERTYPE TO MODIFY
    Jvmserver(DFH$JVMS)
    Definesource(THJVM)
    Definetime(07/08/13 11:39:44)
    Changetime(07/09/13 10:32:42)
    Changeusrid(HTS0201)
    Changeagent(Csdapi)
    Changeagrel(0680)

    That being the case how should I capture the dump ? - thanks

    Everything points to a bug in the module that reads and parses the JVM profile for CICS TS v5.1 Developer trial. We can't reproduce the behaviour in our development CICS regions, so please bear with us while we get access to the Developer Trial code and setup a system to reproduce the failure.

    As Dave says, if you are able to set SJ/AP all trace, then enable the JVMSERVER and take a snap dump just after the failure - it would help us immensely. I believe Developer trial is created by taking a snapshot of service-code. The dump should tell us which APARs were in force when your build of Developer Trial was taken, and therefore help us understand which level of code you are running.

    thx

    Ivan

  • IvanHargreaves
    IvanHargreaves
    27 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-09T18:32:36Z  

    Everything points to a bug in the module that reads and parses the JVM profile for CICS TS v5.1 Developer trial. We can't reproduce the behaviour in our development CICS regions, so please bear with us while we get access to the Developer Trial code and setup a system to reproduce the failure.

    As Dave says, if you are able to set SJ/AP all trace, then enable the JVMSERVER and take a snap dump just after the failure - it would help us immensely. I believe Developer trial is created by taking a snapshot of service-code. The dump should tell us which APARs were in force when your build of Developer Trial was taken, and therefore help us understand which level of code you are running.

    thx

    Ivan

    Hi David,

    We've managed to run with the same value for JVMPROFILEDIR on the same version of Developer Trial as you report (expiring 12/09/2013) - and still cannot reproduce the error.

    Inspecting the code shows no obvious way that the concatenation of JVMPROFILEDIR and JVMPROFILE can result in the missing JVMPROFILEDIR portion of the string (/DFHTSGI).

    While we continue to investigate...can you confirm that the LE runtime options (DFHAXRO) is the CICS supplied version, and not one you have compiled yourself? (I seem to recall another thread where you were tuning various options). If it is your own version, please could you re-run with the CICS version just to rule out LE storage changes.

    thx

    Ivan

  • DavidHarvey
    DavidHarvey
    10 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-10T09:42:24Z  

    Hi David,

    We've managed to run with the same value for JVMPROFILEDIR on the same version of Developer Trial as you report (expiring 12/09/2013) - and still cannot reproduce the error.

    Inspecting the code shows no obvious way that the concatenation of JVMPROFILEDIR and JVMPROFILE can result in the missing JVMPROFILEDIR portion of the string (/DFHTSGI).

    While we continue to investigate...can you confirm that the LE runtime options (DFHAXRO) is the CICS supplied version, and not one you have compiled yourself? (I seem to recall another thread where you were tuning various options). If it is your own version, please could you re-run with the CICS version just to rule out LE storage changes.

    thx

    Ivan

    Hi Ivan, I changed back to the supplied version of DFHAXRO and get the

    'CEE0814S Insufficient storage was available to extend the stack.' error.

    I get this consistently with the supplied DFHAXRO. Interestingly, I sometimes get this with my altered version of DFHAXRO - when I don't get the

    '2013/07/10 08:00:24.973339 GMT Entry: read_profile JVM Profile = /DFHTSGI
    2013/07/10 08:00:25.163191 GMT Event: Changing to = tmp'

    error.

    So I maybe some kind of storage issue which I am changing in some way, sometimes with my version on the DFHAXRO module. - Here is my ulimit -a response:

    HTS0201:TMVS(TEC):/TMVS/export/home/hts0201 > ulimit -a
    core file         16344b
    cpu time          7803
    data size         unlimited
    file size         unlimited
    stack size        unlimited
    file descriptors  64000
    address space     420852k
    memory above bar  3072m
    HTS0201:TMVS(TEC):/TMVS/export/home/hts0201 >

    I run with differing MEMLIMIT values I have tried 20G and 200G and have an EDSALIM of 1400M. So not sure if this issue maybe ZFS related in some way. Another thing I just noticed is that sometimes when I am using my version of DFHAXRO and I get the :

    2013/07/10 08:00:24.973339 GMT Entry: read_profile JVM Profile = /DFHTSGI
    2013/07/10 08:00:25.163191 GMT Event: Changing to = tmp

    and the

    2013/07/10 08:00:25.044173 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file or directory.

    message I also get this message in /TMVS/tmp/AE3BCITH.DFH$JVMS.dfhjvmtrc :

    JVMSERVER DFH$JVMS initializing - Date: 10 Jul 2013 Time: 08:00:25 GMT PID: 0000066907
    2013/07/10 08:00:25.382775 GMT Entry: ipt_control_loop

    thanks

    David


     

  • IvanHargreaves
    IvanHargreaves
    27 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-10T10:47:57Z  

    Hi Ivan, I changed back to the supplied version of DFHAXRO and get the

    'CEE0814S Insufficient storage was available to extend the stack.' error.

    I get this consistently with the supplied DFHAXRO. Interestingly, I sometimes get this with my altered version of DFHAXRO - when I don't get the

    '2013/07/10 08:00:24.973339 GMT Entry: read_profile JVM Profile = /DFHTSGI
    2013/07/10 08:00:25.163191 GMT Event: Changing to = tmp'

    error.

    So I maybe some kind of storage issue which I am changing in some way, sometimes with my version on the DFHAXRO module. - Here is my ulimit -a response:

    HTS0201:TMVS(TEC):/TMVS/export/home/hts0201 > ulimit -a
    core file         16344b
    cpu time          7803
    data size         unlimited
    file size         unlimited
    stack size        unlimited
    file descriptors  64000
    address space     420852k
    memory above bar  3072m
    HTS0201:TMVS(TEC):/TMVS/export/home/hts0201 >

    I run with differing MEMLIMIT values I have tried 20G and 200G and have an EDSALIM of 1400M. So not sure if this issue maybe ZFS related in some way. Another thing I just noticed is that sometimes when I am using my version of DFHAXRO and I get the :

    2013/07/10 08:00:24.973339 GMT Entry: read_profile JVM Profile = /DFHTSGI
    2013/07/10 08:00:25.163191 GMT Event: Changing to = tmp

    and the

    2013/07/10 08:00:25.044173 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file or directory.

    message I also get this message in /TMVS/tmp/AE3BCITH.DFH$JVMS.dfhjvmtrc :

    JVMSERVER DFH$JVMS initializing - Date: 10 Jul 2013 Time: 08:00:25 GMT PID: 0000066907
    2013/07/10 08:00:25.382775 GMT Entry: ipt_control_loop

    thanks

    David


     

    Hi David,

    Yes I think the altered DFHAXRO is causing the issue. The supplied DFHAXRO is a very 'general' set of values which cover most applications. In your case it sounds like you are running a specialised JNI application which is very stack intensive, so let me take a look at tuning it for your specific case.

    Rather than draw out this thread with too much detail, would you be happy to e-mail me directly ivanh@uk.ibm.com. Then when you're happy - we'll post the solution to the thread for posterity.

    thx

    Ivan

  • DavidHarvey
    DavidHarvey
    10 Posts

    Re: File not found when starting JVM Server

    ‏2013-07-11T14:13:28Z  

    Hi David,

    Yes I think the altered DFHAXRO is causing the issue. The supplied DFHAXRO is a very 'general' set of values which cover most applications. In your case it sounds like you are running a specialised JNI application which is very stack intensive, so let me take a look at tuning it for your specific case.

    Rather than draw out this thread with too much detail, would you be happy to e-mail me directly ivanh@uk.ibm.com. Then when you're happy - we'll post the solution to the thread for posterity.

    thx

    Ivan

    Hi Ivan, Thanks for your help. This is now resolved. As discussd, The SDFJAUTH library had the wrong version of DFHSJSCL. We use a rotation of dataset names - PA, PB, PC etc and when moving these between LPARs,  it seems SDFJAUTH contents were not updated to the trial version. I'm just glad to get this resolved and thanks again for your help