Example of Default Precision Rules

This example shows how the default precision rules work.

DName+++++++++++ETDsFrom+++To/L+++IDc.Keywords+++++++++++++++++++++++++++
D FLD1            S             15P 4
D FLD2            S             15P 2
D FLD3            S              5P 2
D FLD4            S              9P 4
D FLD5            S              9P 4
CL0N01Factor1+++++++Opcode(E)+Extended-factor2+++++++++++++++++++++++++++
C                   EVAL      FLD1 = FLD2/(((FLD3/100)*FLD4)+FLD5)
                                            (    1   )
                                           (          2    )
                                          (                 3    )
                                    (     4                       )

When the above Calculation specification is processed, the resulting value assigned to FLD1 will have a precision of zero decimals, not the three decimals expected. The reason is that when it gets to the last evaluation ( 4  in the above example), the number to which the factor is scaled is negative. To see why, look at how the expression is evaluated.

 1 

Evaluate FLD3/100

Rules:

 Lr = 63 
 Dr = max(63-((L1-D1)+D2),0)    
    = max(63-((5-2)+0),0)    
    = max(63-3,0)    
    = 60 

 2 

Evaluate (Result of 1 * FLD4)

Rules:

Lr = min(L1+L2,63)    
   = min(63+9,63)    
   = 63 
Dr = min(D1+D2,63-min((L1-D1)+(L2-D2),63))    
   = min(60+4,63-min((63-60)+(9-4),63))    
   = min(64,63-min(4+5,63)    
   = min(64,55)    
   = 55 

 3 

Evaluate (Result of 2 + FLD5)

Rules:

T  = min(max(L1-D1,L2-D2)+1,63)    
   = min(max(63-55,9-4)+1,63)    
   = min(max(8,5)+1,63)    
   = min(9,63)    
   = 9 
Dr = min(max(D1,D2),31-T)    
   = min(max(55,4),63-9)    
   = min(55,54)    
   = 54 
Lr = T + Dr    
   = 9 + 54 = 63 

 4 

Evaluate FLD2/Result of 3

Rules:

Lr = 63 
Dr = max(63-((L1-D1)+D2),0)    
   = max(63-((15-2)+ 54),0)    
   = max(63-(13+54),0)    
   = max(-4,0)     
****  NEGATIVE NUMBER TO WHICH FACTOR IS SCALED   ****    = 0

To avoid this problem, you can change the above expression so that the first evaluation is a multiplication rather than a division, that is, FLD3 * 0.01 or use the %DEC built-in function to set the sub-expression FLD3/100: %DEC(FLD3/100 : 15 : 4) or use operation extender (R) to ensure that the number of decimal positions never falls below 4.