Topic
4 replies Latest Post - ‏2012-02-29T16:35:18Z by barbara_morris
SystemAdmin
SystemAdmin
535 Posts
ACCEPTED ANSWER

Pinned topic Math not correct in free form calculation

‏2012-02-28T18:56:00Z |
i5/OS = V5R4

I have the following RPGLE code:

d totqty          s             13s 4 inz d totamt          s             15s 4 inz d KHEQFT          s              5s 0 inz d xft             s             13s 4 inz d avgcst          s             13s 4 inz   totamt = 395.1300; xft    = 126.0000; KHEQFT = 100;   avgcst = totamt/(xft/KHEQFT);  
// avgcst = 313.0000   totqty = xft/KHEQFT; avgcst = totamt / totqty;      
// acgcst = 313.5952


The result of avgcst in 'avgcst = totamt/(xft/KHEQFT)' is 313.0000 (Incorrect)

When I made the statement in to two statements:
totqty = xft/KHEQFT;
avgcst = totamt / totqty;
then the result of avgcst is 313.5952. (Correct)

Not sure if I am missing a PTF or something.

Thanks,
Howard
Updated on 2012-02-29T16:35:18Z at 2012-02-29T16:35:18Z by barbara_morris
  • SystemAdmin
    SystemAdmin
    535 Posts
    ACCEPTED ANSWER

    Re: Math not correct in free form calculation

    ‏2012-02-28T21:20:29Z  in response to SystemAdmin
    I think this catches many people by surprise. The traditional fixed-form calculations didn't have any intermediate results - factor 1, factor 2 and the result field were all defined. In contrast, consider 5 * 3 / 2. The compiler multiplies 5 and 3 and stores that intermediate result in a work field. You the programmer don't define this work field; the compiler does it for you. This can be a new environment for many RPG programmers.

    The rules for the intermediate fields and the precision of the calculations can be found here: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/books_web/c0925086551.htm#HDRPRCSNR Precision Rules for Numeric Operations (under Expressions, under Operations, Expressions and Functions in the RPG Reference).
    • barbara_morris
      barbara_morris
      341 Posts
      ACCEPTED ANSWER

      Re: Math not correct in free form calculation

      ‏2012-02-28T21:35:11Z  in response to SystemAdmin
      Howard, there's a few things you can do to correct this
      • Multiply by .01 instead of dividing by 100
      • Avoid code like (a / b / c) that involves dividing a division, by using two statements
      • Use %DECH to tell the compiler how to define the temporary for (b / c): avgcst = totamt/%dech(xft/KHEQFT:13:4)
      • Code the (R) extender on the EVAL, to cause it to use the "result decimal positions" rule
      • Code EXPROPTS(*RESDECPOS) on your H spec

      But I do recommend that you read the documentation that Buck posted, so you understand the ramifications of each one of these alternatives.
      • SystemAdmin
        SystemAdmin
        535 Posts
        ACCEPTED ANSWER

        Re: Math not correct in free form calculation

        ‏2012-02-29T15:41:18Z  in response to barbara_morris
        Hi Barbara and Buck,

        Thank you both for the valuable information. After reading the documentation Buck posted and Barbara's examples, I am gonna to use the following as a general rule:

        1) EXPROPTS(*RESDECPOS) on H specs
        2) use eval(m) when necessary
        3) use %dech where appropriate

        These should work for most cases ? According to the documentation, RPG supports numeric values 'only' up to 63 digits, we should have enough digits to use in most calculations without need to drop any decimal digits .

        Thanks again.

        Happy Leap Day,
        Howard
        • barbara_morris
          barbara_morris
          341 Posts
          ACCEPTED ANSWER

          Re: Math not correct in free form calculation

          ‏2012-02-29T16:35:18Z  in response to SystemAdmin
          That sounds like a good plan.

          I think that it's possible to do any decimal calculation in RPG as long as more than 63 digits of precision isn't required. Since 63 digits is almost sufficient to count all the atoms in the universe, it's hard to imagine a scenario where more than 63 is needed.

          With sufficient test scenarios that test both normal values and extreme values, you should be able to find the correct way to do each calculation.