Topic
13 replies Latest Post - ‏2013-05-16T05:38:29Z by TuukkaIlomäki
SystemAdmin
SystemAdmin
6195 Posts
ACCEPTED ANSWER

Pinned topic MCH0601 on iSeries - how to solve?

‏2013-04-02T12:30:24Z |
Our web architecture is based on calling iSeries programs using jt400. It has worked fine, but this mogring we started getting MCH0601 errors. The crashing program is particularly simple.
program WA49W type basicProgram (TYPARAIN TYPARAIN, TYPARBOUT TYPARBOUT, wA49WEIN WA49WEIN, wA49WFOUT WA49WFOUT)
 
        function main()
                call "WA49S" (tYPARAIN, tYPARBOUT, wA49WEIN.WA49SEIN, wA49WFOUT.WA49SFOUT);
        end
 
end

It is just a wrapper program that calls another program. We have used the problematic program for months but today it started generating MCH0601 errors. But what makes it even more strange is that if I add a couple of lines for logging purposes, the problem disappears.
program WA49W type basicProgram (TYPARAIN TYPARAIN, TYPARBOUT TYPARBOUT, wA49WEIN WA49WEIN, wA49WFOUT WA49WFOUT)
 
        function main()
                SysLib.writeStdout("Program WA49W calling WA49S");
                call "WA49S" (tYPARAIN, tYPARBOUT, wA49WEIN.WA49SEIN, wA49WFOUT.WA49SFOUT);
                SysLib.writeStdout("Program WA49W done calling WA49S");
        end
 
end

There are no relevant lines in iSeries job log (except those generated by Syslib.writeStdout). Googling for MCH0601 does not return any useful information. As far as I know nothing has been changed in our iSeries server (but you never know). Any ideas how to troubleshoot this? Or where to look for more information?
Updated on 2014-03-25T04:35:13Z at 2014-03-25T04:35:13Z by iron-man
  • nick_tn
    nick_tn
    583 Posts
    ACCEPTED ANSWER

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-02T12:45:13Z  in response to SystemAdmin
    could you provide the definitions for the parms being passed in your egl program, also the same for the program you are calling on the iseries.

    Also, have you compiled the iseries program recently.
    • markevans
      markevans
      2758 Posts
      ACCEPTED ANSWER

      Re: MCH0601 on iSeries - how to solve?

      ‏2013-04-02T14:15:14Z  in response to nick_tn
      Hi,
      With the MCH0601, you should have a message in the job log on what program it was in and the offset within that program.

      Did anything change..such as the level of QEGL or did you regenerate the programs that are involved in the calling sequence?

      Did you get both writestdout messages? If so, that means it is taking a machine check on termination.

      A couple of suggestions/comments:

      a.) Does anything in the WA49S program (or what it calls) fill in an array. I have seen programs write more elements of an array than are defined..which in turn writes over storage... and strange things occur. Generating with CheckIndeces=yes will catch these with an IndexOutofBounds exception, but this option does add performance overhead. VAGen did not have this type of check.

      b.) Generate the calling (and called) program(s) with statementTrace=yes. This should narrow down which paragraph in the program is executing when the MCH occurs. This listing could be attached.

      c.) Attach a compile listing (with offsets if possible) so that someone can look at what was executing and see what may be causing it.

      d.) Open a PMR and send in the docs above so that the support team can take a look.
      • SystemAdmin
        SystemAdmin
        6195 Posts
        ACCEPTED ANSWER

        Re: MCH0601 on iSeries - how to solve?

        ‏2013-04-02T18:56:29Z  in response to markevans
        To Mark and Nick,
        thank you for your suggestions. The parameters are conventional small structured records - we have used similar records in dozens of places. (I learned using level numbers the hard way when figuring out how to call RPG programs as described in this thread. Definitions are at the end of this message.

        The curious thing is that the programs have not been modified and they have been used daily hundreds of times during the past few months. The timestamp of the objects was January 7th, 2013. As far as I know, nothing has been changed on our iSeries server. At least nothing related to EGL. The stream of MCH0601 errors started completely unexpectedly. It must be that something has changed. I am sure about EGL runtime and the programs themselves but I have not idea of what else I should be checking.

        The called program does a couple of inserts (successfully) and commits changes. No arrays are involved. I got both writestdout messages so it appears to be the calling. I will try your other suggestions when I get back to work.

        
        Record TYPARAIN type basicRecord 3 TYPARAIN_RECORD 
        
        char(25) ; 4 TPITPMODE 
        
        char(2) ; 4 TPITSMODE 
        
        char(2) ; 4 TPIITMRTV num(7) ; 4 TPIFCALL 
        
        char(1) ; 4 TPICALMOD 
        
        char(3) ; 4 TPICALPRG 
        
        char(10) ; end   Record TYPARBOUT type basicRecord 3 TYPARBOUT_RECORD 
        
        char(125) ; 4 TPOMSGTPE 
        
        char(3) ; 4 TPOMSGTXT 
        
        char(70) ; 4 TPOMSGFLD 
        
        char(10) ; 4 TPOMSFLIN num(3) ; 4 TPOITMFND 
        
        char(1) ; 4 TPOITMALL 
        
        char(4) ; 4 TPOITMSEL 
        
        char(1) ; 4 TPOITMCNT smallint ; 4 TPOMAPRFS 
        
        char(1) ; 4 TPOCANCEL 
        
        char(1) ; 4 TPOERRDTL 
        
        char(3) ; 4 TYPARBOUT_LOPUT 
        
        char(26) ; end   Record WA49WEIN type basicRecord 3 WA49SEIN WA49SEIN; end   Record WA49SEIN type basicRecord 4 YPTUNNUS 
        
        char(50); 4 YPHASH 
        
        char(128); 4 ASTYYPPI 
        
        char(2); 4 ASNRO num(11); 4 KYNRO num(11); 4 YHNRO num(11); 4 NIMI 
        
        char(50); 4 PWDHASH 
        
        char(128); end   Record WA49WFOUT type basicRecord 3 WA49SFOUT WA49SFOUT; end   Record WA49SFOUT type basicRecord 4 WKTUNNUS 
        
        char(50); 4 YHNRO num(11); end
        
        • SystemAdmin
          SystemAdmin
          6195 Posts
          ACCEPTED ANSWER

          Re: MCH0601 on iSeries - how to solve?

          ‏2013-04-03T10:55:51Z  in response to SystemAdmin
          A few updates. First, I forgot to mention that the problem occurs only on our production server (which makes testing really fun). Second, I was not able to use statementTrace=yes since it is available only for z/OS and we are using IBM i.

          Third, apparently it is the SysLib.writeStdout statement before program call "WA49S" that helps avoiding MCH0601: having only a SysLib.writeStdout statement after the call still resulted in a crash.

          Fourth, the detailed error message is
          Space offset X'00002000' or X'0000000000000000' is outside current limit for object QZRCSRVS  QUSER     195630.
          
          The program is qvgnextf and the module is csodtsi in case those are relevant.

          Finally, our iSeries maintenance staff confirmed that no updates (such as PTFs) have been applied to the system. The only change I can think of is DST but that is a little far fetched. I might still try finding a quiet moment in our services and killing all QZRCSRVS jobs (WA49W is invoked from a web service through jt400) - we have had recently a case in which that helped solve sudden crash in a third party software application.
          Updated on 2014-03-25T04:35:24Z at 2014-03-25T04:35:24Z by iron-man
          • Jeff.Douglas
            Jeff.Douglas
            194 Posts
            ACCEPTED ANSWER

            Re: MCH0601 on iSeries - how to solve?

            ‏2013-04-03T14:13:56Z  in response to SystemAdmin
            Terve Tuukka.

            Minun nimeni Jeff.

            Mark asked me to take a look at your problem. When reading through this, I noticed that the machine check is occurring in CSODTSI. This module is very old and has not been used since version 6. I don't understand why your cobol code would be accessing this, without seeing the CBL listing from your EGL program.

            Can you please send us the CBL listing for this program, both with and without the writestdout statements that you have added. A PMR would be the best route to do this with. Please open one.

            Something to note: CSODTSI was a C program that called the C runtime to obtain the date/time/timestamp/number formatting values from the C runtime. This code has been replaced by VGNDTSI which is written in COBOL and calls the LE runtime for these same formatting characters. If your C runtime has changed, that might explain the sudden machine checks.
            • markevans
              markevans
              2758 Posts
              ACCEPTED ANSWER

              Re: MCH0601 on iSeries - how to solve?

              ‏2013-04-03T14:17:15Z  in response to Jeff.Douglas
              Thanks Jeff.

              Also, just so you know, statementTrace=yes does work for iSeries as well. It will write its output to the Joblog that is running under QZRSRVCS. (same log as writestdout).
            • SystemAdmin
              SystemAdmin
              6195 Posts
              ACCEPTED ANSWER

              Re: MCH0601 on iSeries - how to solve?

              ‏2013-04-04T05:25:58Z  in response to Jeff.Douglas
              Terve Jeff. Paljon kiitoksia.

              I opened a PMR with the generated code. As far as I know, no changes has been applied to our C runtime - we do not write any code for IBM i in C. In case it matters, we do run third party software written using VAGen.

              To Mark. First of all, thanks for getting Jeff involved. Secondly, regarding statementTrace. RBD does not allow me to set the statementTrace option in the build descriptor. However, then I tried adding statementTrace="YES" to build descriptor with text editor and lo and behold, the generated COBOL contained dozens of lines of type DISPLAY "WA49W: ENTERED PROCEDURE EZECONTROL". Hence, the manual says you cannot do it and there is no UI for it, but you can do it.
              • markevans
                markevans
                2758 Posts
                ACCEPTED ANSWER

                Re: MCH0601 on iSeries - how to solve?

                ‏2013-04-04T14:20:25Z  in response to SystemAdmin
                Tuukka,

                The problem is that statementTrace does not appear in the iSeries filter. If you change the filter to "all" then it will appear in the UI (see attached from RBD V7.5) and you can change it. This is a valid way to work..

                We need to get the filter updated so that statementTrace is included in the ones for iSeries and update the doc.

                And are you saying that you have a mix of EGL and VAGen generated code in this scenario?

                Would you also have a mix of EGL V6 and EGL V7+ generated COBOL?
                • SystemAdmin
                  SystemAdmin
                  6195 Posts
                  ACCEPTED ANSWER

                  Re: MCH0601 on iSeries - how to solve?

                  ‏2013-04-04T18:24:58Z  in response to markevans
                  Mark,
                  in the present scenario that is causing the trouble (WA49W calling WA49S) we have only EGL V7 generated COBOL programs. Both applications were written and generated with RBD 7.5.1.8 and we even did not use any of the V6 compatibility features of the language.

                  We do have a lot of legacy V6 applications and in other programs we call EGL V6 applications - it takes some time to regenerate and test about 1000 programs. We also call third party VAGen programs from our EGL V7 programs. And we call legacy V6 programs that call those VAGen programs. And of course, the spaghetti would not be complete without some hand-written COBOL programs and integration with RPG programs developed with Plex.
                  • markevans
                    markevans
                    2758 Posts
                    ACCEPTED ANSWER

                    Re: MCH0601 on iSeries - how to solve?

                    ‏2013-04-05T14:23:47Z  in response to SystemAdmin
                    Tuukka,

                    Thanks. The reason I asked was that we have had some issues with mixing of VAGen and EGL runtime activation groups when the generated code from both are used within one call stack. It happens fairly randomly and many people run the mixed environment successfully. That said, we have a change for it if needed...but this does not sound like your case.
                • TuukkaIlomäki
                  TuukkaIlomäki
                  67 Posts
                  ACCEPTED ANSWER

                  Re: MCH0601 on iSeries - how to solve?

                  ‏2013-04-15T18:50:26Z  in response to markevans

                  Hello Mark, I attached a picture of my build options and it does not show statementTrace even if filter is set to All. Hence, I had to set the build option using a text editor. 

                  Attachments

                  • markevans
                    markevans
                    2758 Posts
                    ACCEPTED ANSWER

                    Re: MCH0601 on iSeries - how to solve?

                    ‏2013-04-16T15:26:06Z  in response to TuukkaIlomäki

                    We figured it out.  The statementTrace options is added to the EGL Build descriptor editor when you install the Generation for system z feature (which most likely is not available for RDI SOA).    Given you are only doing iSeries, I assume you did not install that feature (understandably). even if it was available. 

                    Starting in V8.0.1, there is no longer a feature for generation on system z.  It is automatically enabled by the standard RBD install.

                    We will open a a defect to get it added for iSeriesC and in the filter as well.

  • TuukkaIlomäki
    TuukkaIlomäki
    67 Posts
    ACCEPTED ANSWER

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-05-16T05:38:29Z  in response to SystemAdmin

    A patched version of QVGNEXTF solved the issue. Thanks to the developer for quick and efficient help. Sinä tiedät kuka sinä olet. :)