Topic
14 replies Latest Post - ‏2013-07-11T14:13:28Z by DavidHarvey
DavidHarvey
DavidHarvey
10 Posts
ACCEPTED ANSWER

Pinned topic File not found when starting JVM Server

‏2013-07-04T14:49:16Z |

I get the following error whenever I try to start a JVM in the CICS TS 5.1 Developer Trial version.

2013/07/04 07:58:48.438702 GMT *Exc*: Attempt to open JVM profile DFHOSGI has failed. Runtime error message = EDC5129I No such file
or directory.

 

The JVMPROFILEDIR=points to the directory containing the JVM Profile and USSHOME is also correct. This happens no matter what profile I try to use. The JVMtrace, JVMin and JVMout are all empty only out put is that from the dfhaxro module in jvmerr:

  

Storage Report for Enclave CELQPIPI Thu Jul  4 08:13:34 2013
Language Environment V01 R13.00

STACK64 statistics:
Initial size:                                               200M
Increment size:                                             100M
Maximum used by all concurrent threads:                     222M
Largest used by any thread:                                 222M
Number of increments allocated:                               1
64bit User HEAP statistics:
Initial size:                                               200M
Increment size:                                               4M

 

Not sure what is wrong with my set-up...

 

.

  • IvanHargreaves
    IvanHargreaves
    25 Posts
    ACCEPTED ANSWER

    Re: File not found when starting JVM Server

    ‏2013-07-05T08:48:38Z  in response to DavidHarvey

    Hi David,

    The 8 char JVMPROFILE on your JVMSERVER resource must match exactly a file found in the JVMPROFILEDIR (specified in your SIT). It is a unix file, so it is case-sensitive. In CICS TS v5.1 the file must not have any extension. If you look in the JVMPROFILEDIR for that region and see a file called "DFHOSGI" then please check that the CICS userid has authority to access the file. If your CICS region id has access to read that file (you can use ls -al from a USS shell to check) then there must be something more fundamental.

    If you continue to have problems, set SJ and AP domain trace to 'ALL', and turn on auxtrace before enabling the JVMSERVER. If you can then get the cics auxtrace, the CICS job output, along with dfhjvmerr, dfhjvmout, and dfhjvmtrc if they exist, to me. I'll take a look.

    thx

    Ivan

    • DavidHarvey
      DavidHarvey
      10 Posts
      ACCEPTED ANSWER

      Re: File not found when starting JVM Server

      ‏2013-07-05T11:28:13Z  in response to IvanHargreaves

      Ivan - thanks for looking at this - I've attached some doc for you - I was using my own JVMProfiles Directory but find that even the supplied directory has this issue. The dfhjvmerr,dfhjvmout and dfhjvmtrc exist in the /tmp/ directory but are all empty. The attached trace is from the third attemt to enable the jvm in this run of CICS - so this time stamp:

       

      2013/07/05 10:29:05.613078 GMT *Exc*: Attempt to open JVM profile DFHOSGI has failed. Runtime error message = EDC5129I No such file
      or directory.

      Alos go this output one time when using the spplied JVMProfile directoy:

      2013/07/05 10:29:05.575583 GMT Entry: read_profile JVM Profile = 0/DFHOSGI
      2013/07/05 10:29:05.627150 GMT Event: Changing to = tmp

       

      Attachments

      • IvanHargreaves
        IvanHargreaves
        25 Posts
        ACCEPTED ANSWER

        Re: File not found when starting JVM Server

        ‏2013-07-05T15:33:33Z  in response to DavidHarvey

        The message 0/DFHOSGI indicates a problem with the JVMPROFILEDIR. I notice you are pointing at the JVMProfiles directory in the CICS install tree. I would expect that location to be read-only? (which would explain the problem). However, if you are sure your CICS installation directories are no longer read-only then I will have to take a deeper look at the trace files provided.

        thx

        Ivan

        • DavidHarvey
          DavidHarvey
          10 Posts
          ACCEPTED ANSWER

          Re: File not found when starting JVM Server

          ‏2013-07-09T10:01:52Z  in response to IvanHargreaves

          Hi Ivan, I have run the test again with a different ZFS directory where the JVMProfiles are stored. I'm attaching 1) the CICS log for THCICT13, 2) the AUXTRACE and 3) some info regarding the ZFS permissions etc. Please let me know if you need anything else - thanks

          • DavidHarvey
            DavidHarvey
            10 Posts
            ACCEPTED ANSWER

            Re: File not found when starting JVM Server

            ‏2013-07-09T10:27:04Z  in response to DavidHarvey

             

            Attachments

          • DavidRoberts
            DavidRoberts
            2 Posts
            ACCEPTED ANSWER

            Re: File not found when starting JVM Server

            ‏2013-07-09T12:07:12Z  in response to DavidHarvey

            Hi David,

            With SJ and AP domain trace set to 'ALL' we would expect to see the trace from the enable of the JVM server in a file in the CICS job log.  The output is written to stdout so will be in the SYSPRINT DD if specified, or a SYSnnnnn generated DD.  The first line in the file should be something like:

            2013/07/09 11:52:48.039432 UTC DFHOSGI Entry: read_profile() - JVM Profile  = /u/cicsuser/JVMProfiles/DFHOSGI
            

            The trace we are looking for is in this file, not the aux trace.  Please check your trace settings again, enable the JVM server and check the CICS job log for the trace output.

            Thanks

            Dave

            • DavidHarvey
              DavidHarvey
              10 Posts
              ACCEPTED ANSWER

              Re: File not found when starting JVM Server

              ‏2013-07-09T12:18:08Z  in response to DavidRoberts

              Hi Dave - I cut and pasted these into the attachment 'ZFS Permissions and USERIDs etc.docx' above SYS00002 ans SYS0003 appeared when attempting to enable the JVM - these also appear in the job log for THCICT13(THCICT13.TXT -above) - as shown here:

                DFHFC0201 07/09/2013 10:43:34 AE3BCITH Non-RLS file DFHCSD has been deallocated. Module DFHFCN.
               2013/07/09 09:32:55.104637 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file
               or directory.
               2013/07/09 09:36:32.616090 GMT Entry: read_profile JVM Profile = /DFHTSGI
               2013/07/09 09:36:32.653832 GMT Event: Changing to = tmp
               2013/07/09 09:36:32.641004 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file
               or directory.

               

              Let me know if you need anything else - thanks

              • DavidRoberts
                DavidRoberts
                2 Posts
                ACCEPTED ANSWER

                Re: File not found when starting JVM Server

                ‏2013-07-09T12:48:37Z  in response to DavidHarvey

                OK, thanks.

                So, the strange thing is why your log shows /DFHTSGI rather than /TMVS/u/tstcict/JVMProfiles/DFHTSGI.  The message is created by concatenating JVMPROFILEDIR "/" JVMPROFILE.  So the JVMPROFILEDIR has somehow gone missing.

                So, to try and track down what's happening, please CEMT INQUIRE your JVM server.  The profile directory is shown in the expanded view.  If this looks OK, we probably need a dump to investigate further

                Thanks,

                Dave

                 

                • DavidHarvey
                  DavidHarvey
                  10 Posts
                  ACCEPTED ANSWER

                  Re: File not found when starting JVM Server

                  ‏2013-07-09T13:01:47Z  in response to DavidRoberts

                  Dave,

                  The expaned Cemt I JVM shows this :

                  I JVM
                  RESULT - OVERTYPE TO MODIFY
                  Jvmserver(DFH$JVMS)
                  Enablestatus( Disabled )
                  Purgetype(              )
                  Prfile(DFHTSGI)
                  Lerunopts(DFHAXRO)
                  Threadcount(000)
                  Threadlimit( 015 )
                  Currentheap(0)
                  Initheap(0)
                  Maxheap(0)
                  Gcpolicy()
                  Occupancy(0)
                  Pid(0000000000)
                  Profiledir(/TMVS/u/tstcict/JVMProfiles/)
                  Installtime(07/09/13 13:43:16)
                  Installusrid(HTS0201)
                  Installagent(Csdapi)
                  Definesource(THJVM)

                  I JVM
                  RESULT - OVERTYPE TO MODIFY
                  Jvmserver(DFH$JVMS)
                  Definesource(THJVM)
                  Definetime(07/08/13 11:39:44)
                  Changetime(07/09/13 10:32:42)
                  Changeusrid(HTS0201)
                  Changeagent(Csdapi)
                  Changeagrel(0680)

                  That being the case how should I capture the dump ? - thanks

                  • IvanHargreaves
                    IvanHargreaves
                    25 Posts
                    ACCEPTED ANSWER

                    Re: File not found when starting JVM Server

                    ‏2013-07-09T17:08:37Z  in response to DavidHarvey

                    Everything points to a bug in the module that reads and parses the JVM profile for CICS TS v5.1 Developer trial. We can't reproduce the behaviour in our development CICS regions, so please bear with us while we get access to the Developer Trial code and setup a system to reproduce the failure.

                    As Dave says, if you are able to set SJ/AP all trace, then enable the JVMSERVER and take a snap dump just after the failure - it would help us immensely. I believe Developer trial is created by taking a snapshot of service-code. The dump should tell us which APARs were in force when your build of Developer Trial was taken, and therefore help us understand which level of code you are running.

                    thx

                    Ivan

                    • IvanHargreaves
                      IvanHargreaves
                      25 Posts
                      ACCEPTED ANSWER

                      Re: File not found when starting JVM Server

                      ‏2013-07-09T18:32:36Z  in response to IvanHargreaves

                      Hi David,

                      We've managed to run with the same value for JVMPROFILEDIR on the same version of Developer Trial as you report (expiring 12/09/2013) - and still cannot reproduce the error.

                      Inspecting the code shows no obvious way that the concatenation of JVMPROFILEDIR and JVMPROFILE can result in the missing JVMPROFILEDIR portion of the string (/DFHTSGI).

                      While we continue to investigate...can you confirm that the LE runtime options (DFHAXRO) is the CICS supplied version, and not one you have compiled yourself? (I seem to recall another thread where you were tuning various options). If it is your own version, please could you re-run with the CICS version just to rule out LE storage changes.

                      thx

                      Ivan

                      • DavidHarvey
                        DavidHarvey
                        10 Posts
                        ACCEPTED ANSWER

                        Re: File not found when starting JVM Server

                        ‏2013-07-10T09:42:24Z  in response to IvanHargreaves

                        Hi Ivan, I changed back to the supplied version of DFHAXRO and get the

                        'CEE0814S Insufficient storage was available to extend the stack.' error.

                        I get this consistently with the supplied DFHAXRO. Interestingly, I sometimes get this with my altered version of DFHAXRO - when I don't get the

                        '2013/07/10 08:00:24.973339 GMT Entry: read_profile JVM Profile = /DFHTSGI
                        2013/07/10 08:00:25.163191 GMT Event: Changing to = tmp'

                        error.

                        So I maybe some kind of storage issue which I am changing in some way, sometimes with my version on the DFHAXRO module. - Here is my ulimit -a response:

                        HTS0201:TMVS(TEC):/TMVS/export/home/hts0201 > ulimit -a
                        core file         16344b
                        cpu time          7803
                        data size         unlimited
                        file size         unlimited
                        stack size        unlimited
                        file descriptors  64000
                        address space     420852k
                        memory above bar  3072m
                        HTS0201:TMVS(TEC):/TMVS/export/home/hts0201 >

                        I run with differing MEMLIMIT values I have tried 20G and 200G and have an EDSALIM of 1400M. So not sure if this issue maybe ZFS related in some way. Another thing I just noticed is that sometimes when I am using my version of DFHAXRO and I get the :

                        2013/07/10 08:00:24.973339 GMT Entry: read_profile JVM Profile = /DFHTSGI
                        2013/07/10 08:00:25.163191 GMT Event: Changing to = tmp

                        and the

                        2013/07/10 08:00:25.044173 GMT *Exc*: Attempt to open JVM profile DFHTSGI has failed. Runtime error message = EDC5129I No such file or directory.

                        message I also get this message in /TMVS/tmp/AE3BCITH.DFH$JVMS.dfhjvmtrc :

                        JVMSERVER DFH$JVMS initializing - Date: 10 Jul 2013 Time: 08:00:25 GMT PID: 0000066907
                        2013/07/10 08:00:25.382775 GMT Entry: ipt_control_loop

                        thanks

                        David


                         

                        • IvanHargreaves
                          IvanHargreaves
                          25 Posts
                          ACCEPTED ANSWER

                          Re: File not found when starting JVM Server

                          ‏2013-07-10T10:47:57Z  in response to DavidHarvey

                          Hi David,

                          Yes I think the altered DFHAXRO is causing the issue. The supplied DFHAXRO is a very 'general' set of values which cover most applications. In your case it sounds like you are running a specialised JNI application which is very stack intensive, so let me take a look at tuning it for your specific case.

                          Rather than draw out this thread with too much detail, would you be happy to e-mail me directly ivanh@uk.ibm.com. Then when you're happy - we'll post the solution to the thread for posterity.

                          thx

                          Ivan

                          • DavidHarvey
                            DavidHarvey
                            10 Posts
                            ACCEPTED ANSWER

                            Re: File not found when starting JVM Server

                            ‏2013-07-11T14:13:28Z  in response to IvanHargreaves

                            Hi Ivan, Thanks for your help. This is now resolved. As discussd, The SDFJAUTH library had the wrong version of DFHSJSCL. We use a rotation of dataset names - PA, PB, PC etc and when moving these between LPARs,  it seems SDFJAUTH contents were not updated to the trial version. I'm just glad to get this resolved and thanks again for your help