Precision Rules for Numeric Operations

Unlike the fixed-form operation codes where you must always specify the result of each individual operation, RPG must determine the format and precision of the result of each operation within an expression.

If an operation has a result of format float, integer, or unsigned the precision is the maximum size for that format. Integer and unsigned operations produce 4-byte values and float operations produce 8-byte values.

However, if the operation has a packed-decimal, zoned decimal, or binary format, the precision of the result depends on the precisions of the operands.

It is important to be aware of the precision rules for decimal operations since even a relatively simple expression may have a result that may not be what you expect. For example, if the two operands of a multiplication are large enough, the result of the multiplication will have zero decimal places. If you are multiplying two 40 digit numbers, ideally you would need a 80 digit result to hold all possible results of the multiplication. However, since RPG supports numeric values only up to 63 digits, the result is adjusted to 63 digits. In this case, as many as 17 decimal digits are dropped from the result.

There are two sets of precision rules that you can use to control the sizes of intermediate values:

1. The default rules give you intermediate results that are as large as possible in order to minimize the possibility of numeric overflow. Unfortunately, in certain cases, this may yield results with zero decimal places if the result is very large.

   For more information on the default precision rules, see:

   • Using the Default Precision Rules
   • Precision of Intermediate Results
   • Example of Default Precision Rules
2. The "Result Decimal Positions" precision rule works the same as the default rule except that if the statement involves an assignment to a numeric variable or a conversion to a specific decimal precision, the number of decimal positions of any intermediate result is never reduced below the desired result decimal places.

   In practice, you don't have to worry about the exact precisions if you examine the compile listing when coding numeric expressions. A diagnostic message indicates that decimal positions are being dropped in an intermediate result. If there is an assignment involved in the expression, you can ensure that the decimal positions are kept by using the "Result Decimal Positions" precision rule for the statement by coding operation code extender (R).

   If the "Result Decimal Position" precision rule cannot be used (say, in a relational expression), built-in function %DEC can be used to convert the result of a sub-expression to a smaller precision which may prevent the decimal positions from being lost.

   For more information on the "Result Decimal Positions" precision rules, see:

   • Using the "Result Decimal Position" Precision Rules
   • Example of "Result Decimal Position" Precision Rules