Topic
  • 13 replies
  • Latest Post - ‏2013-05-16T05:38:29Z by TuukkaIlomäki
SystemAdmin
SystemAdmin
6195 Posts

Pinned topic MCH0601 on iSeries - how to solve?

‏2013-04-02T12:30:24Z |
Our web architecture is based on calling iSeries programs using jt400. It has worked fine, but this mogring we started getting MCH0601 errors. The crashing program is particularly simple.
program WA49W type basicProgram (TYPARAIN TYPARAIN, TYPARBOUT TYPARBOUT, wA49WEIN WA49WEIN, wA49WFOUT WA49WFOUT)
 
        function main()
                call "WA49S" (tYPARAIN, tYPARBOUT, wA49WEIN.WA49SEIN, wA49WFOUT.WA49SFOUT);
        end
 
end

It is just a wrapper program that calls another program. We have used the problematic program for months but today it started generating MCH0601 errors. But what makes it even more strange is that if I add a couple of lines for logging purposes, the problem disappears.
program WA49W type basicProgram (TYPARAIN TYPARAIN, TYPARBOUT TYPARBOUT, wA49WEIN WA49WEIN, wA49WFOUT WA49WFOUT)
 
        function main()
                SysLib.writeStdout("Program WA49W calling WA49S");
                call "WA49S" (tYPARAIN, tYPARBOUT, wA49WEIN.WA49SEIN, wA49WFOUT.WA49SFOUT);
                SysLib.writeStdout("Program WA49W done calling WA49S");
        end
 
end

There are no relevant lines in iSeries job log (except those generated by Syslib.writeStdout). Googling for MCH0601 does not return any useful information. As far as I know nothing has been changed in our iSeries server (but you never know). Any ideas how to troubleshoot this? Or where to look for more information?
Updated on 2014-03-25T04:35:13Z at 2014-03-25T04:35:13Z by iron-man
  • nick_tn
    nick_tn
    584 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-02T12:45:13Z  
    could you provide the definitions for the parms being passed in your egl program, also the same for the program you are calling on the iseries.

    Also, have you compiled the iseries program recently.
  • markevans
    markevans
    2843 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-02T14:15:14Z  
    • nick_tn
    • ‏2013-04-02T12:45:13Z
    could you provide the definitions for the parms being passed in your egl program, also the same for the program you are calling on the iseries.

    Also, have you compiled the iseries program recently.
    Hi,
    With the MCH0601, you should have a message in the job log on what program it was in and the offset within that program.

    Did anything change..such as the level of QEGL or did you regenerate the programs that are involved in the calling sequence?

    Did you get both writestdout messages? If so, that means it is taking a machine check on termination.

    A couple of suggestions/comments:

    a.) Does anything in the WA49S program (or what it calls) fill in an array. I have seen programs write more elements of an array than are defined..which in turn writes over storage... and strange things occur. Generating with CheckIndeces=yes will catch these with an IndexOutofBounds exception, but this option does add performance overhead. VAGen did not have this type of check.

    b.) Generate the calling (and called) program(s) with statementTrace=yes. This should narrow down which paragraph in the program is executing when the MCH occurs. This listing could be attached.

    c.) Attach a compile listing (with offsets if possible) so that someone can look at what was executing and see what may be causing it.

    d.) Open a PMR and send in the docs above so that the support team can take a look.
  • SystemAdmin
    SystemAdmin
    6195 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-02T18:56:29Z  
    • markevans
    • ‏2013-04-02T14:15:14Z
    Hi,
    With the MCH0601, you should have a message in the job log on what program it was in and the offset within that program.

    Did anything change..such as the level of QEGL or did you regenerate the programs that are involved in the calling sequence?

    Did you get both writestdout messages? If so, that means it is taking a machine check on termination.

    A couple of suggestions/comments:

    a.) Does anything in the WA49S program (or what it calls) fill in an array. I have seen programs write more elements of an array than are defined..which in turn writes over storage... and strange things occur. Generating with CheckIndeces=yes will catch these with an IndexOutofBounds exception, but this option does add performance overhead. VAGen did not have this type of check.

    b.) Generate the calling (and called) program(s) with statementTrace=yes. This should narrow down which paragraph in the program is executing when the MCH occurs. This listing could be attached.

    c.) Attach a compile listing (with offsets if possible) so that someone can look at what was executing and see what may be causing it.

    d.) Open a PMR and send in the docs above so that the support team can take a look.
    To Mark and Nick,
    thank you for your suggestions. The parameters are conventional small structured records - we have used similar records in dozens of places. (I learned using level numbers the hard way when figuring out how to call RPG programs as described in this thread. Definitions are at the end of this message.

    The curious thing is that the programs have not been modified and they have been used daily hundreds of times during the past few months. The timestamp of the objects was January 7th, 2013. As far as I know, nothing has been changed on our iSeries server. At least nothing related to EGL. The stream of MCH0601 errors started completely unexpectedly. It must be that something has changed. I am sure about EGL runtime and the programs themselves but I have not idea of what else I should be checking.

    The called program does a couple of inserts (successfully) and commits changes. No arrays are involved. I got both writestdout messages so it appears to be the calling. I will try your other suggestions when I get back to work.

    
    Record TYPARAIN type basicRecord 3 TYPARAIN_RECORD 
    
    char(25) ; 4 TPITPMODE 
    
    char(2) ; 4 TPITSMODE 
    
    char(2) ; 4 TPIITMRTV num(7) ; 4 TPIFCALL 
    
    char(1) ; 4 TPICALMOD 
    
    char(3) ; 4 TPICALPRG 
    
    char(10) ; end   Record TYPARBOUT type basicRecord 3 TYPARBOUT_RECORD 
    
    char(125) ; 4 TPOMSGTPE 
    
    char(3) ; 4 TPOMSGTXT 
    
    char(70) ; 4 TPOMSGFLD 
    
    char(10) ; 4 TPOMSFLIN num(3) ; 4 TPOITMFND 
    
    char(1) ; 4 TPOITMALL 
    
    char(4) ; 4 TPOITMSEL 
    
    char(1) ; 4 TPOITMCNT smallint ; 4 TPOMAPRFS 
    
    char(1) ; 4 TPOCANCEL 
    
    char(1) ; 4 TPOERRDTL 
    
    char(3) ; 4 TYPARBOUT_LOPUT 
    
    char(26) ; end   Record WA49WEIN type basicRecord 3 WA49SEIN WA49SEIN; end   Record WA49SEIN type basicRecord 4 YPTUNNUS 
    
    char(50); 4 YPHASH 
    
    char(128); 4 ASTYYPPI 
    
    char(2); 4 ASNRO num(11); 4 KYNRO num(11); 4 YHNRO num(11); 4 NIMI 
    
    char(50); 4 PWDHASH 
    
    char(128); end   Record WA49WFOUT type basicRecord 3 WA49SFOUT WA49SFOUT; end   Record WA49SFOUT type basicRecord 4 WKTUNNUS 
    
    char(50); 4 YHNRO num(11); end
    
  • SystemAdmin
    SystemAdmin
    6195 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-03T10:55:51Z  
    To Mark and Nick,
    thank you for your suggestions. The parameters are conventional small structured records - we have used similar records in dozens of places. (I learned using level numbers the hard way when figuring out how to call RPG programs as described in this thread. Definitions are at the end of this message.

    The curious thing is that the programs have not been modified and they have been used daily hundreds of times during the past few months. The timestamp of the objects was January 7th, 2013. As far as I know, nothing has been changed on our iSeries server. At least nothing related to EGL. The stream of MCH0601 errors started completely unexpectedly. It must be that something has changed. I am sure about EGL runtime and the programs themselves but I have not idea of what else I should be checking.

    The called program does a couple of inserts (successfully) and commits changes. No arrays are involved. I got both writestdout messages so it appears to be the calling. I will try your other suggestions when I get back to work.

    <pre class="jive-pre"> Record TYPARAIN type basicRecord 3 TYPARAIN_RECORD char(25) ; 4 TPITPMODE char(2) ; 4 TPITSMODE char(2) ; 4 TPIITMRTV num(7) ; 4 TPIFCALL char(1) ; 4 TPICALMOD char(3) ; 4 TPICALPRG char(10) ; end Record TYPARBOUT type basicRecord 3 TYPARBOUT_RECORD char(125) ; 4 TPOMSGTPE char(3) ; 4 TPOMSGTXT char(70) ; 4 TPOMSGFLD char(10) ; 4 TPOMSFLIN num(3) ; 4 TPOITMFND char(1) ; 4 TPOITMALL char(4) ; 4 TPOITMSEL char(1) ; 4 TPOITMCNT smallint ; 4 TPOMAPRFS char(1) ; 4 TPOCANCEL char(1) ; 4 TPOERRDTL char(3) ; 4 TYPARBOUT_LOPUT char(26) ; end Record WA49WEIN type basicRecord 3 WA49SEIN WA49SEIN; end Record WA49SEIN type basicRecord 4 YPTUNNUS char(50); 4 YPHASH char(128); 4 ASTYYPPI char(2); 4 ASNRO num(11); 4 KYNRO num(11); 4 YHNRO num(11); 4 NIMI char(50); 4 PWDHASH char(128); end Record WA49WFOUT type basicRecord 3 WA49SFOUT WA49SFOUT; end Record WA49SFOUT type basicRecord 4 WKTUNNUS char(50); 4 YHNRO num(11); end </pre>
    A few updates. First, I forgot to mention that the problem occurs only on our production server (which makes testing really fun). Second, I was not able to use statementTrace=yes since it is available only for z/OS and we are using IBM i.

    Third, apparently it is the SysLib.writeStdout statement before program call "WA49S" that helps avoiding MCH0601: having only a SysLib.writeStdout statement after the call still resulted in a crash.

    Fourth, the detailed error message is
    Space offset X'00002000' or X'0000000000000000' is outside current limit for object QZRCSRVS  QUSER     195630.
    
    The program is qvgnextf and the module is csodtsi in case those are relevant.

    Finally, our iSeries maintenance staff confirmed that no updates (such as PTFs) have been applied to the system. The only change I can think of is DST but that is a little far fetched. I might still try finding a quiet moment in our services and killing all QZRCSRVS jobs (WA49W is invoked from a web service through jt400) - we have had recently a case in which that helped solve sudden crash in a third party software application.
    Updated on 2014-03-25T04:35:24Z at 2014-03-25T04:35:24Z by iron-man
  • Jeff.Douglas
    Jeff.Douglas
    209 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-03T14:13:56Z  
    A few updates. First, I forgot to mention that the problem occurs only on our production server (which makes testing really fun). Second, I was not able to use statementTrace=yes since it is available only for z/OS and we are using IBM i.

    Third, apparently it is the SysLib.writeStdout statement before program call "WA49S" that helps avoiding MCH0601: having only a SysLib.writeStdout statement after the call still resulted in a crash.

    Fourth, the detailed error message is <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr">Space offset X'00002000' or X'0000000000000000' is outside current limit for object QZRCSRVS QUSER 195630. </pre> The program is qvgnextf and the module is csodtsi in case those are relevant.

    Finally, our iSeries maintenance staff confirmed that no updates (such as PTFs) have been applied to the system. The only change I can think of is DST but that is a little far fetched. I might still try finding a quiet moment in our services and killing all QZRCSRVS jobs (WA49W is invoked from a web service through jt400) - we have had recently a case in which that helped solve sudden crash in a third party software application.
    Terve Tuukka.

    Minun nimeni Jeff.

    Mark asked me to take a look at your problem. When reading through this, I noticed that the machine check is occurring in CSODTSI. This module is very old and has not been used since version 6. I don't understand why your cobol code would be accessing this, without seeing the CBL listing from your EGL program.

    Can you please send us the CBL listing for this program, both with and without the writestdout statements that you have added. A PMR would be the best route to do this with. Please open one.

    Something to note: CSODTSI was a C program that called the C runtime to obtain the date/time/timestamp/number formatting values from the C runtime. This code has been replaced by VGNDTSI which is written in COBOL and calls the LE runtime for these same formatting characters. If your C runtime has changed, that might explain the sudden machine checks.
  • markevans
    markevans
    2843 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-03T14:17:15Z  
    Terve Tuukka.

    Minun nimeni Jeff.

    Mark asked me to take a look at your problem. When reading through this, I noticed that the machine check is occurring in CSODTSI. This module is very old and has not been used since version 6. I don't understand why your cobol code would be accessing this, without seeing the CBL listing from your EGL program.

    Can you please send us the CBL listing for this program, both with and without the writestdout statements that you have added. A PMR would be the best route to do this with. Please open one.

    Something to note: CSODTSI was a C program that called the C runtime to obtain the date/time/timestamp/number formatting values from the C runtime. This code has been replaced by VGNDTSI which is written in COBOL and calls the LE runtime for these same formatting characters. If your C runtime has changed, that might explain the sudden machine checks.
    Thanks Jeff.

    Also, just so you know, statementTrace=yes does work for iSeries as well. It will write its output to the Joblog that is running under QZRSRVCS. (same log as writestdout).
  • SystemAdmin
    SystemAdmin
    6195 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-04T05:25:58Z  
    Terve Tuukka.

    Minun nimeni Jeff.

    Mark asked me to take a look at your problem. When reading through this, I noticed that the machine check is occurring in CSODTSI. This module is very old and has not been used since version 6. I don't understand why your cobol code would be accessing this, without seeing the CBL listing from your EGL program.

    Can you please send us the CBL listing for this program, both with and without the writestdout statements that you have added. A PMR would be the best route to do this with. Please open one.

    Something to note: CSODTSI was a C program that called the C runtime to obtain the date/time/timestamp/number formatting values from the C runtime. This code has been replaced by VGNDTSI which is written in COBOL and calls the LE runtime for these same formatting characters. If your C runtime has changed, that might explain the sudden machine checks.
    Terve Jeff. Paljon kiitoksia.

    I opened a PMR with the generated code. As far as I know, no changes has been applied to our C runtime - we do not write any code for IBM i in C. In case it matters, we do run third party software written using VAGen.

    To Mark. First of all, thanks for getting Jeff involved. Secondly, regarding statementTrace. RBD does not allow me to set the statementTrace option in the build descriptor. However, then I tried adding statementTrace="YES" to build descriptor with text editor and lo and behold, the generated COBOL contained dozens of lines of type DISPLAY "WA49W: ENTERED PROCEDURE EZECONTROL". Hence, the manual says you cannot do it and there is no UI for it, but you can do it.
  • markevans
    markevans
    2843 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-04T14:20:25Z  
    Terve Jeff. Paljon kiitoksia.

    I opened a PMR with the generated code. As far as I know, no changes has been applied to our C runtime - we do not write any code for IBM i in C. In case it matters, we do run third party software written using VAGen.

    To Mark. First of all, thanks for getting Jeff involved. Secondly, regarding statementTrace. RBD does not allow me to set the statementTrace option in the build descriptor. However, then I tried adding statementTrace="YES" to build descriptor with text editor and lo and behold, the generated COBOL contained dozens of lines of type DISPLAY "WA49W: ENTERED PROCEDURE EZECONTROL". Hence, the manual says you cannot do it and there is no UI for it, but you can do it.
    Tuukka,

    The problem is that statementTrace does not appear in the iSeries filter. If you change the filter to "all" then it will appear in the UI (see attached from RBD V7.5) and you can change it. This is a valid way to work..

    We need to get the filter updated so that statementTrace is included in the ones for iSeries and update the doc.

    And are you saying that you have a mix of EGL and VAGen generated code in this scenario?

    Would you also have a mix of EGL V6 and EGL V7+ generated COBOL?
  • SystemAdmin
    SystemAdmin
    6195 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-04T18:24:58Z  
    • markevans
    • ‏2013-04-04T14:20:25Z
    Tuukka,

    The problem is that statementTrace does not appear in the iSeries filter. If you change the filter to "all" then it will appear in the UI (see attached from RBD V7.5) and you can change it. This is a valid way to work..

    We need to get the filter updated so that statementTrace is included in the ones for iSeries and update the doc.

    And are you saying that you have a mix of EGL and VAGen generated code in this scenario?

    Would you also have a mix of EGL V6 and EGL V7+ generated COBOL?
    Mark,
    in the present scenario that is causing the trouble (WA49W calling WA49S) we have only EGL V7 generated COBOL programs. Both applications were written and generated with RBD 7.5.1.8 and we even did not use any of the V6 compatibility features of the language.

    We do have a lot of legacy V6 applications and in other programs we call EGL V6 applications - it takes some time to regenerate and test about 1000 programs. We also call third party VAGen programs from our EGL V7 programs. And we call legacy V6 programs that call those VAGen programs. And of course, the spaghetti would not be complete without some hand-written COBOL programs and integration with RPG programs developed with Plex.
  • markevans
    markevans
    2843 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-05T14:23:47Z  
    Mark,
    in the present scenario that is causing the trouble (WA49W calling WA49S) we have only EGL V7 generated COBOL programs. Both applications were written and generated with RBD 7.5.1.8 and we even did not use any of the V6 compatibility features of the language.

    We do have a lot of legacy V6 applications and in other programs we call EGL V6 applications - it takes some time to regenerate and test about 1000 programs. We also call third party VAGen programs from our EGL V7 programs. And we call legacy V6 programs that call those VAGen programs. And of course, the spaghetti would not be complete without some hand-written COBOL programs and integration with RPG programs developed with Plex.
    Tuukka,

    Thanks. The reason I asked was that we have had some issues with mixing of VAGen and EGL runtime activation groups when the generated code from both are used within one call stack. It happens fairly randomly and many people run the mixed environment successfully. That said, we have a change for it if needed...but this does not sound like your case.
  • TuukkaIlomäki
    TuukkaIlomäki
    68 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-15T18:50:26Z  
    • markevans
    • ‏2013-04-04T14:20:25Z
    Tuukka,

    The problem is that statementTrace does not appear in the iSeries filter. If you change the filter to "all" then it will appear in the UI (see attached from RBD V7.5) and you can change it. This is a valid way to work..

    We need to get the filter updated so that statementTrace is included in the ones for iSeries and update the doc.

    And are you saying that you have a mix of EGL and VAGen generated code in this scenario?

    Would you also have a mix of EGL V6 and EGL V7+ generated COBOL?

    Hello Mark, I attached a picture of my build options and it does not show statementTrace even if filter is set to All. Hence, I had to set the build option using a text editor. 

    Attachments

  • markevans
    markevans
    2843 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-04-16T15:26:06Z  

    Hello Mark, I attached a picture of my build options and it does not show statementTrace even if filter is set to All. Hence, I had to set the build option using a text editor. 

    We figured it out.  The statementTrace options is added to the EGL Build descriptor editor when you install the Generation for system z feature (which most likely is not available for RDI SOA).    Given you are only doing iSeries, I assume you did not install that feature (understandably). even if it was available. 

    Starting in V8.0.1, there is no longer a feature for generation on system z.  It is automatically enabled by the standard RBD install.

    We will open a a defect to get it added for iSeriesC and in the filter as well.

  • TuukkaIlomäki
    TuukkaIlomäki
    68 Posts

    Re: MCH0601 on iSeries - how to solve?

    ‏2013-05-16T05:38:29Z  

    A patched version of QVGNEXTF solved the issue. Thanks to the developer for quick and efficient help. Sinä tiedät kuka sinä olet. :)