Topic
  • 10 replies
  • Latest Post - ‏2012-04-24T15:51:14Z by SystemAdmin
SystemAdmin
SystemAdmin
535 Posts

Pinned topic Automatic storage overflow - using V6R1 with large variables in procedures

‏2012-04-18T20:19:23Z |
With V6R1, the possibility of using larger variables were introduced (this was a great alternative to using pointers). I have begun using them, but ran into an issue today where I'm receiving a "Automatic storage overflow" error.

I basically have several generic procedures which have large input/output variables (some are around 2MB - Varying(4)). If I make use of some of these procedures I have no problems, but at some point when I call enough of these procedures in 1 program I get an "Automatic storage overflow" error during execution. In my current example, the error ISN'T actually being shown with the line number of the newly added procedure call, but at another line where I'm calling that same procedure.

Am I using these large variables incorrectly? Can I compile my program differently to allow it to work? Is there something I can do to my job to allow this to work?

Any help would be appreciated.

Thanks,
Dave Weissman
Updated on 2012-04-24T15:51:14Z at 2012-04-24T15:51:14Z by SystemAdmin
  • scott_klement
    scott_klement
    245 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-18T21:30:50Z  
    Automatic storage is "temporary storage" that's on the call stack. In RPG, it's typically used for variables that are defined to be local to a subprocedure. (This includes parameters if they are passed by VALUE, or returned by value.)

    In the normal/default/traditional storage model (*SNGLVL), you can have approximately 16 MB of total automatic storage in a job. For emphasis, I'll repeat that: 16 mb total for a job. (It's not 16 mb per variable, like it would be on the heap or in static storage.)

    (That limit is MUCH higher if you use the *TERASPACE storage model... but that requires IBM i 7.1)

    So each time you call a pgm/proc the automatic (non-static) variables for that pgm/proc are created by reserving automatic storage for them on the call stack. Each time the pgm/proc ends, that memory is, of course, freed. But, if the pgm/proc calls another pgm/proc before it ends, then both are on the call stack at the same time, and both of them are using some of the call stack's automatic storage. It's not unusual for a job to have a call stack with 15-20 pgms/procs all on the stack at the same time. And the issue is the total of automatic storage used by ALL of them. Then the total exceeds approx 16mb, you get the MCH4429 "Automatic Storage Overflow".

    Nothing changed about this in 6.1. Except, of course, that it's now easier to declare larger variables, possibly making it more likely that you'll exceed that 16mb limit.

    How can you solve the problem? By changing some of the variables to be static (add the STATIC keyword), or by using pointers to allocate them dynamically off the heap, or by (yuck) converting them to global variables. On IBM i 7.1, of course, you can also use STGMDL(*TERASPACE) which makes the limit much higher.

    Make sense? Not sure if I explained this very well... it's kinda tough, since it has to do with the way the operating system works, something we don't normally have to worry about.
  • SystemAdmin
    SystemAdmin
    535 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-23T12:49:24Z  
    Automatic storage is "temporary storage" that's on the call stack. In RPG, it's typically used for variables that are defined to be local to a subprocedure. (This includes parameters if they are passed by VALUE, or returned by value.)

    In the normal/default/traditional storage model (*SNGLVL), you can have approximately 16 MB of total automatic storage in a job. For emphasis, I'll repeat that: 16 mb total for a job. (It's not 16 mb per variable, like it would be on the heap or in static storage.)

    (That limit is MUCH higher if you use the *TERASPACE storage model... but that requires IBM i 7.1)

    So each time you call a pgm/proc the automatic (non-static) variables for that pgm/proc are created by reserving automatic storage for them on the call stack. Each time the pgm/proc ends, that memory is, of course, freed. But, if the pgm/proc calls another pgm/proc before it ends, then both are on the call stack at the same time, and both of them are using some of the call stack's automatic storage. It's not unusual for a job to have a call stack with 15-20 pgms/procs all on the stack at the same time. And the issue is the total of automatic storage used by ALL of them. Then the total exceeds approx 16mb, you get the MCH4429 "Automatic Storage Overflow".

    Nothing changed about this in 6.1. Except, of course, that it's now easier to declare larger variables, possibly making it more likely that you'll exceed that 16mb limit.

    How can you solve the problem? By changing some of the variables to be static (add the STATIC keyword), or by using pointers to allocate them dynamically off the heap, or by (yuck) converting them to global variables. On IBM i 7.1, of course, you can also use STGMDL(*TERASPACE) which makes the limit much higher.

    Make sense? Not sure if I explained this very well... it's kinda tough, since it has to do with the way the operating system works, something we don't normally have to worry about.
    Scott - thanks for the quick reply. I tried to respond Wednesday night, but the site appeared to have issues and I didn't have a chance to get back to it until now.

    I have used the "static" keyword with java related programming, but didn't realized it could be used with traditional RPG procedures. I will Google-It to get some additional information, but if you know of resources can you please share the links?

    What's happening in my situation is odd in that my program works up to a point, until I add a call to another procedure (with large input/output parameters). After adding that procedure call, the program throws the error at an EARLIER line rather than this new one. Using the prototype below as an example (which is the additional procedure taht I'm calling), besides changing the definition to have smaller variable sizes (or swapping to pointers), would I be better off eliminating the output declaration and instead make the 1st lookin parameter passed by reference to handle both input and output? Would that have an impact on temporary storage?

    D @replace PR A LEN(2097152) VARYING(4)
    D lookin A LEN(2097152) VARYING(4)
    D OPTIONS(*VARSIZE) CONST
    D lookfor 256 VALUE VARYING
    D replwth A LEN(2097152) VARYING(4)
    D OPTIONS(*VARSIZE) CONST
  • barbara_morris
    barbara_morris
    393 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-23T14:03:26Z  
    Scott - thanks for the quick reply. I tried to respond Wednesday night, but the site appeared to have issues and I didn't have a chance to get back to it until now.

    I have used the "static" keyword with java related programming, but didn't realized it could be used with traditional RPG procedures. I will Google-It to get some additional information, but if you know of resources can you please share the links?

    What's happening in my situation is odd in that my program works up to a point, until I add a call to another procedure (with large input/output parameters). After adding that procedure call, the program throws the error at an EARLIER line rather than this new one. Using the prototype below as an example (which is the additional procedure taht I'm calling), besides changing the definition to have smaller variable sizes (or swapping to pointers), would I be better off eliminating the output declaration and instead make the 1st lookin parameter passed by reference to handle both input and output? Would that have an impact on temporary storage?

    D @replace PR A LEN(2097152) VARYING(4)
    D lookin A LEN(2097152) VARYING(4)
    D OPTIONS(*VARSIZE) CONST
    D lookfor 256 VALUE VARYING
    D replwth A LEN(2097152) VARYING(4)
    D OPTIONS(*VARSIZE) CONST
    Dave, there was a compiler bug causing MCH4429, related to CONST parameters where a concatenated value was passed to the parameter. Make sure you're update on RPG compiler PTFs.

    The latest 6.1 compiler PTF is SI46499. The latest 7.1 compiler PTF is SI46561. The latest 7.1 PTF for TGTRLS(V6R1M0) is SI45902.

    But it might just be that you have too much automatic storage in one or more of your procedures. Defining large local variables with the STATIC keyword is one way to reduce the amount of static storage. Just a warning: With STATIC, the variable is only initialized once. You might have to add an assignment statement or a CLEAR statement at the beginning of the procedure to get the initialization re-done for each call.

    The procedure that is failing is not necessarily the one with the "large" amount of automatic storage. It might be that some earlier procedure got the level of automatic storage near to the 16MB maximum, and the procedure getting the error just tipped it over the edge. It might also be the procedure being called that has the large amount of automatic storage, and a small amount of automatic storage added to the calling procedure meant that the called procedure couldn't be called.

    I probably didn't explain that very well. Let me try again ... say the limit was 16. Your call stack has P1 -> P2 -> P3 -> P4.

    A working case: P1 uses 1, P2 uses 12, P3 uses 2, P4 uses 1. When P3 is running, the auto storage total is 1 + 12 + 2 = 15. P3 can successfully call P4 because P4 uses 1, so the new total will be 16, still within the limit.

    Failing case 1: P2 increases to 13. Now when P3 is on the stack, the auto storage is already at 16. P3 cannot call P4 because that would put the total to 17.

    Failing case 2: P4 increases to 2. When P3 is on the stack, the limit is 15, but P3 can't call P4 because that would put the total to 17.

    Failing case 3: P3 increases to 3. Same as failing case 1.

    For any given module, you can see the largest automatic storage required for a procedure in the module by using DSPMOD DETAIL(*SIZE) and page down to the page that starts with "Procedure size (decompressed) and limit information:". It will show which procedure in the module that uses the most automatic storage.

    If you're a bit lucky, the procedure that's getting the error will show up there, and the amount of auto storage will be large (near 16MB). I say "lucky" because you only have to work on that procedure to reduce your auto storage. If it doesn't seem very high, you'll have to look at your call stack at the time of the failure (DSPJOB OPTION(*PGMSTK) and
    hunt down the problem procedure by doing DSPMOD on all the modules that contain your procedures.

    If one of the modules doesn't show your particular procedure as having the largest auto storage, it means your procedure has less auto storage than the one that's listed. Depending on how big that amount is, you may or may not want to recompile a temporary version of that module that only has that one procedure.

    Finally, be aware that the total automatic storage required for a procedure isn't just the automatic variables that you declare. There's also some auto storage defined as temporary variables by the compiler, and there's also some auto storage defined by the system to handle your calls.

    I realize this post is probably too long but I might as well tell everything I can think of that might help to solve your problem.

    Oh, one more thing :) The automatic storage used by the program stack goes up and down as procedures are called and return. Say procedure PROC1 requires 100 bytes of auto storage. If the total auto storage for the program stack is 1000 while PROC1 is running. When PROC1 returns, the total will go back down to 1000 - 100 = 900.
  • SystemAdmin
    SystemAdmin
    535 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-23T16:24:27Z  
    Dave, there was a compiler bug causing MCH4429, related to CONST parameters where a concatenated value was passed to the parameter. Make sure you're update on RPG compiler PTFs.

    The latest 6.1 compiler PTF is SI46499. The latest 7.1 compiler PTF is SI46561. The latest 7.1 PTF for TGTRLS(V6R1M0) is SI45902.

    But it might just be that you have too much automatic storage in one or more of your procedures. Defining large local variables with the STATIC keyword is one way to reduce the amount of static storage. Just a warning: With STATIC, the variable is only initialized once. You might have to add an assignment statement or a CLEAR statement at the beginning of the procedure to get the initialization re-done for each call.

    The procedure that is failing is not necessarily the one with the "large" amount of automatic storage. It might be that some earlier procedure got the level of automatic storage near to the 16MB maximum, and the procedure getting the error just tipped it over the edge. It might also be the procedure being called that has the large amount of automatic storage, and a small amount of automatic storage added to the calling procedure meant that the called procedure couldn't be called.

    I probably didn't explain that very well. Let me try again ... say the limit was 16. Your call stack has P1 -> P2 -> P3 -> P4.

    A working case: P1 uses 1, P2 uses 12, P3 uses 2, P4 uses 1. When P3 is running, the auto storage total is 1 + 12 + 2 = 15. P3 can successfully call P4 because P4 uses 1, so the new total will be 16, still within the limit.

    Failing case 1: P2 increases to 13. Now when P3 is on the stack, the auto storage is already at 16. P3 cannot call P4 because that would put the total to 17.

    Failing case 2: P4 increases to 2. When P3 is on the stack, the limit is 15, but P3 can't call P4 because that would put the total to 17.

    Failing case 3: P3 increases to 3. Same as failing case 1.

    For any given module, you can see the largest automatic storage required for a procedure in the module by using DSPMOD DETAIL(*SIZE) and page down to the page that starts with "Procedure size (decompressed) and limit information:". It will show which procedure in the module that uses the most automatic storage.

    If you're a bit lucky, the procedure that's getting the error will show up there, and the amount of auto storage will be large (near 16MB). I say "lucky" because you only have to work on that procedure to reduce your auto storage. If it doesn't seem very high, you'll have to look at your call stack at the time of the failure (DSPJOB OPTION(*PGMSTK) and
    hunt down the problem procedure by doing DSPMOD on all the modules that contain your procedures.

    If one of the modules doesn't show your particular procedure as having the largest auto storage, it means your procedure has less auto storage than the one that's listed. Depending on how big that amount is, you may or may not want to recompile a temporary version of that module that only has that one procedure.

    Finally, be aware that the total automatic storage required for a procedure isn't just the automatic variables that you declare. There's also some auto storage defined as temporary variables by the compiler, and there's also some auto storage defined by the system to handle your calls.

    I realize this post is probably too long but I might as well tell everything I can think of that might help to solve your problem.

    Oh, one more thing :) The automatic storage used by the program stack goes up and down as procedures are called and return. Say procedure PROC1 requires 100 bytes of auto storage. If the total auto storage for the program stack is 1000 while PROC1 is running. When PROC1 returns, the total will go back down to 1000 - 100 = 900.
    Barbara - thanks for the reply. We do not have the PTF installed so we'll be ordering it to see if that gets us any further. The procedure causing the problem is the one showing up under the heading "procedure size (decompressed...".

    Is declaring a large local variable with the Static keyword the same as having a global variable?

    For the time being I have reduced the size of a few procedures and now all is well (of course they have a lower limit now, but still should be sufficient).
  • barbara_morris
    barbara_morris
    393 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-23T20:29:20Z  
    Barbara - thanks for the reply. We do not have the PTF installed so we'll be ordering it to see if that gets us any further. The procedure causing the problem is the one showing up under the heading "procedure size (decompressed...".

    Is declaring a large local variable with the Static keyword the same as having a global variable?

    For the time being I have reduced the size of a few procedures and now all is well (of course they have a lower limit now, but still should be sufficient).
    Declaring a local variable with the STATIC keyword is the same as declaring a global variable from the point of view of the type of storage.

    But declaring it local means that only that one subprocedure has access to it, so it's better to keep it local from a maintenance point of view.
  • scott_klement
    scott_klement
    245 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-23T23:05:56Z  
    Declaring a local variable with the STATIC keyword is the same as declaring a global variable from the point of view of the type of storage.

    But declaring it local means that only that one subprocedure has access to it, so it's better to keep it local from a maintenance point of view.
    Barbara,

    Given Dave's example:

    
    D @replace        PR              A   LEN(2097152) VARYING(4) D  lookin                         A   LEN(2097152) VARYING(4) D                                     OPTIONS(*VARSIZE) CONST D  lookfor                     256    VALUE VARYING D  replwth                        A   LEN(2097152) VARYING(4) D                                     OPTIONS(*VARSIZE) CONST
    


    Can you give us an example of how much automatic storage will be used when calling that prototype? Does the compiler use automatic storage for the temporaries that it creates for the CONST parameters? How about the VALUE parameter... I'm sure that uses 258 bytes of automatic storage in the '@replace' procedure, but is there any additional automatic storage used on the caller's end?

    Just wondering.
  • SystemAdmin
    SystemAdmin
    535 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-24T01:07:45Z  
    Declaring a local variable with the STATIC keyword is the same as declaring a global variable from the point of view of the type of storage.

    But declaring it local means that only that one subprocedure has access to it, so it's better to keep it local from a maintenance point of view.
    good idea...that makes sense to me...I never thought about it that way. Like I said before I've never used the static keyword before for traditional (non-java) procedures but now I'll look into the possibility of using it.
  • SystemAdmin
    SystemAdmin
    535 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-24T01:11:42Z  
    Barbara,

    Given Dave's example:

    <pre class="jive-pre"> D @replace PR A LEN(2097152) VARYING(4) D lookin A LEN(2097152) VARYING(4) D OPTIONS(*VARSIZE) CONST D lookfor 256 VALUE VARYING D replwth A LEN(2097152) VARYING(4) D OPTIONS(*VARSIZE) CONST </pre>

    Can you give us an example of how much automatic storage will be used when calling that prototype? Does the compiler use automatic storage for the temporaries that it creates for the CONST parameters? How about the VALUE parameter... I'm sure that uses 258 bytes of automatic storage in the '@replace' procedure, but is there any additional automatic storage used on the caller's end?

    Just wondering.
    Yes - I too am interested in the impact of this function. I know in V7R1 there's a %SCANRPL BIF but until we upgrade I need to use a function like this. Are there better alternatives then writing my own procedure to serve a generic purpose?
  • barbara_morris
    barbara_morris
    393 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-24T15:24:40Z  
    Barbara,

    Given Dave's example:

    <pre class="jive-pre"> D @replace PR A LEN(2097152) VARYING(4) D lookin A LEN(2097152) VARYING(4) D OPTIONS(*VARSIZE) CONST D lookfor 256 VALUE VARYING D replwth A LEN(2097152) VARYING(4) D OPTIONS(*VARSIZE) CONST </pre>

    Can you give us an example of how much automatic storage will be used when calling that prototype? Does the compiler use automatic storage for the temporaries that it creates for the CONST parameters? How about the VALUE parameter... I'm sure that uses 258 bytes of automatic storage in the '@replace' procedure, but is there any additional automatic storage used on the caller's end?

    Just wondering.
    I hadn't looked closely enough at the prototype before. I think it is the large return value that is causing the automatic storage to be so high.

    
    D @replace        PR              A   LEN(2097152) VARYING(4) /free 
    // @replace();                                                
    
    return;
    


    The automatic storage needed by a procedure is a combination of the automatic storage coded by the RPG programmer, the automatic storage added by the compiler for its internal variables and temporaries, and the automatic storage added by the system to actually do the calls (and probably for other things that I don't know about).

    When I compile this module without the call, the automatic storage for that procedure is 96.

    When I compile it with the call, the automatic storage is 2097376. A lot of work has been done by the system to reduce the amount of automatic storage needed for return values. At one point it was 7 times the size of the return value; I think they've done very well to get it down to just 1.

    Most of the temporaries generated by the RPG compiler that are associated with the call would be automatic but some would be static. The exact nature of the temporaries would depend on the specific parameters passed.

    Dave, if you really need to return such a large value, you might consider having two versions of your procedure. One that has a return value for convenience in calling, and another that uses an extra parameter instead.

    In V7R1, there's a RTNPARM keyword for PR and PI that will cause the compiler to treat the return value as an extra parm, giving the convenience of a return value for the caller, while reducing the automatic storage requirements for the "return value", and often increasing the speed of returning the value, especially for large VARYING return values.
  • SystemAdmin
    SystemAdmin
    535 Posts

    Re: Automatic storage overflow - using V6R1 with large variables in procedures

    ‏2012-04-24T15:51:14Z  
    I hadn't looked closely enough at the prototype before. I think it is the large return value that is causing the automatic storage to be so high.

    <pre class="jive-pre"> D @replace PR A LEN(2097152) VARYING(4) /free // @replace(); return; </pre>

    The automatic storage needed by a procedure is a combination of the automatic storage coded by the RPG programmer, the automatic storage added by the compiler for its internal variables and temporaries, and the automatic storage added by the system to actually do the calls (and probably for other things that I don't know about).

    When I compile this module without the call, the automatic storage for that procedure is 96.

    When I compile it with the call, the automatic storage is 2097376. A lot of work has been done by the system to reduce the amount of automatic storage needed for return values. At one point it was 7 times the size of the return value; I think they've done very well to get it down to just 1.

    Most of the temporaries generated by the RPG compiler that are associated with the call would be automatic but some would be static. The exact nature of the temporaries would depend on the specific parameters passed.

    Dave, if you really need to return such a large value, you might consider having two versions of your procedure. One that has a return value for convenience in calling, and another that uses an extra parameter instead.

    In V7R1, there's a RTNPARM keyword for PR and PI that will cause the compiler to treat the return value as an extra parm, giving the convenience of a return value for the caller, while reducing the automatic storage requirements for the "return value", and often increasing the speed of returning the value, especially for large VARYING return values.
    I have done what you suggested and created another procedure with no return value where the 1st parameter is used as input/output. By doing this I no longer get the error. I suppose with values so large I'm opening myself up to these sorts of issues. There seems to be good things packaged with 7.1 and I know at some point we'll be there (maybe sooner than later).

    I appreciate the help from both you and Scott.

    Thanks,
    Dave