Topic
  • 28 replies
  • Latest Post - ‏2014-01-23T09:02:02Z by Adamarla
jester76
jester76
68 Posts

Pinned topic real in DXL

‏2013-04-24T17:27:26Z |
  • llandale
    llandale
    3005 Posts
    ACCEPTED ANSWER

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-04-24T20:48:29Z  
    • jester76
    • ‏2013-04-24T18:42:59Z

    The "real" type has a lot of precision, but when you print it only 6 decimal digits are displayed.  Some time ago I posted a "fStringOfReal()" function which clumsily handled it; other folks posted more elegant solutions.

    -Louie

  • jester76
    jester76
    68 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-04-24T18:42:59Z  
  • llandale
    llandale
    3005 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-04-24T20:48:29Z  
    • jester76
    • ‏2013-04-24T18:42:59Z

    The "real" type has a lot of precision, but when you print it only 6 decimal digits are displayed.  Some time ago I posted a "fStringOfReal()" function which clumsily handled it; other folks posted more elegant solutions.

    -Louie

  • Wolfgang Uhr
    Wolfgang Uhr
    280 Posts

    Re: real in DXL

    ‏2013-04-25T07:28:43Z  

    As you can see in the attachment (I hope the upload will work):

    I do not know what you are speaking about, but it sounds interesting ...

     

    Attachments

  • jester76
    jester76
    68 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-04-25T15:34:28Z  
    • llandale
    • ‏2013-04-24T20:48:29Z

    The "real" type has a lot of precision, but when you print it only 6 decimal digits are displayed.  Some time ago I posted a "fStringOfReal()" function which clumsily handled it; other folks posted more elegant solutions.

    -Louie

    Thanks LLandale. Your function worked great for me. Sorry I had to post everything in the subject line as site wouldn't let me edit the body of the post.

  • jester76
    jester76
    68 Posts

    Re: real in DXL

    ‏2013-04-25T15:37:51Z  

    As you can see in the attachment (I hope the upload will work):

    I do not know what you are speaking about, but it sounds interesting ...

     

    When you print or assign real to object in DXL it only prints/assigns 6 digits after decimal points. So the function will help you go beyond 6 decimal points.

  • jester76
    jester76
    68 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-06-10T21:58:49Z  
    • llandale
    • ‏2013-04-24T20:48:29Z

    The "real" type has a lot of precision, but when you print it only 6 decimal digits are displayed.  Some time ago I posted a "fStringOfReal()" function which clumsily handled it; other folks posted more elegant solutions.

    -Louie

    Hello,

    I am facinng an issue here. When supplying a value of 0.0000009999 to the function. It returns 0.0000009998. Can you please check why this is happening.

    Thanks

    Jester

  • llandale
    llandale
    3005 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-06-11T15:27:12Z  
    • jester76
    • ‏2013-06-10T21:58:49Z

    Hello,

    I am facinng an issue here. When supplying a value of 0.0000009999 to the function. It returns 0.0000009998. Can you please check why this is happening.

    Thanks

    Jester

    "real" is by nature imprecise. 

    1.  On a computer, [a] "2*2" is not equal to [b] "2^^2" (exponent).  [a] uses integers which are precise, [b] converts to real which is not. 

    2.  You cannot accurately express "1/3" in a real number.  Looks like "0.33333333333333333" which is very close but not quite.

    When I run this code:

    • void Test(real r)
    • { print fStringOfReal(r, 0,true) "\t" fStringOfReal(r, 10,true) "\t[" (r*1000000000000.0) "]\n"
    • }
    • Test(0.00000099)
    • Test(0.000000999)
    • Test(0.0000009999)
    • Test(0.00000099999)

    I get these results (bolding added post results)

    • 0.00000099000000000000004 0.00000099     [990000.000000]
    • 0.00000099900000000000018 0.000000999   [999000.000000]
    • 0.00000099989999999999998 0.0000009998 [999900.000000]  <<-- your example
    • 0.00000099999000000000012 0.0000009999 [999990.000000]

     Perhaps at step [8] in the code I could "round" instead of just "truncating".

    Could you post the URL you found this?  IIRC, someone else responded with a far more elegant solution.

    -Louie

    Updated on 2013-06-11T15:30:59Z at 2013-06-11T15:30:59Z by llandale
  • jester76
    jester76
    68 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-06-11T15:43:33Z  
    • llandale
    • ‏2013-06-11T15:27:12Z

    "real" is by nature imprecise. 

    1.  On a computer, [a] "2*2" is not equal to [b] "2^^2" (exponent).  [a] uses integers which are precise, [b] converts to real which is not. 

    2.  You cannot accurately express "1/3" in a real number.  Looks like "0.33333333333333333" which is very close but not quite.

    When I run this code:

    • void Test(real r)
    • { print fStringOfReal(r, 0,true) "\t" fStringOfReal(r, 10,true) "\t[" (r*1000000000000.0) "]\n"
    • }
    • Test(0.00000099)
    • Test(0.000000999)
    • Test(0.0000009999)
    • Test(0.00000099999)

    I get these results (bolding added post results)

    • 0.00000099000000000000004 0.00000099     [990000.000000]
    • 0.00000099900000000000018 0.000000999   [999000.000000]
    • 0.00000099989999999999998 0.0000009998 [999900.000000]  <<-- your example
    • 0.00000099999000000000012 0.0000009999 [999990.000000]

     Perhaps at step [8] in the code I could "round" instead of just "truncating".

    Could you post the URL you found this?  IIRC, someone else responded with a far more elegant solution.

    -Louie

    Hello,

    Here is the link I found this @.

    https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014582477

    I think rounding would be better then truncating. But the funny thing is that it does not do it with .0000008888 or any other number like this.

    Can you please post the solution with rounding.

    Thanks

    Jester

  • llandale
    llandale
    3005 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-06-11T17:07:03Z  
    • jester76
    • ‏2013-06-11T15:43:33Z

    Hello,

    Here is the link I found this @.

    https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014582477

    I think rounding would be better then truncating. But the funny thing is that it does not do it with .0000008888 or any other number like this.

    Can you please post the solution with rounding.

    Thanks

    Jester

    I'll let you debug it.  May look like:

    • int iLast, iNext
    •                                 // [8] Truncate Decimal Part to max DesiredPrecision
    • DecimalLen = length(DecimalPart)
    • if(DesiredPrecision > 0  and
    •    DesiredPrecision < DecimalLen) // Desired is non-zero, but less than Actual -
    • {  iLast = intOf(DecimalPart[DesiredPrecision-1] "")  // last digit desired
    •    iNext = intOf(DecimalPart[DesiredPrecision] "")    // next digit, ignored
    •    if (iNext >= 5) then iLast++    // Round
    •    DecimalPart = DecimalPart[0:DesiredPrecision-2] iLast ""
    • }

    -Louie

  • jester76
    jester76
    68 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-06-11T18:09:55Z  
    • llandale
    • ‏2013-06-11T17:07:03Z

    I'll let you debug it.  May look like:

    • int iLast, iNext
    •                                 // [8] Truncate Decimal Part to max DesiredPrecision
    • DecimalLen = length(DecimalPart)
    • if(DesiredPrecision > 0  and
    •    DesiredPrecision < DecimalLen) // Desired is non-zero, but less than Actual -
    • {  iLast = intOf(DecimalPart[DesiredPrecision-1] "")  // last digit desired
    •    iNext = intOf(DecimalPart[DesiredPrecision] "")    // next digit, ignored
    •    if (iNext >= 5) then iLast++    // Round
    •    DecimalPart = DecimalPart[0:DesiredPrecision-2] iLast ""
    • }

    -Louie

    Actually it is not the question of rounding. if I supply 0.0000009999 it should return same right. Why is last 9 being converted to 8? That is the real question.

    Update:

    I found the issue:

    if I change the follwoign variable anything more than 25 the behavior changes

      cl_fStringOfRealExponent = 25 // Arbitrary limit of Digits away from decimal point

    Not sure why that is.
     

    Updated on 2013-06-11T19:23:00Z at 2013-06-11T19:23:00Z by jester76
  • llandale
    llandale
    3005 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-06-11T19:25:04Z  
    • jester76
    • ‏2013-06-11T18:09:55Z

    Actually it is not the question of rounding. if I supply 0.0000009999 it should return same right. Why is last 9 being converted to 8? That is the real question.

    Update:

    I found the issue:

    if I change the follwoign variable anything more than 25 the behavior changes

      cl_fStringOfRealExponent = 25 // Arbitrary limit of Digits away from decimal point

    Not sure why that is.
     

    It isn't. 

    • 0.0000009999   is being converted to 0.00000099989999999999998, and then truncated to 0.0000009998.
    • 0.00000099999 is being converted to 0.00000099999000000000012, and then truncated to 0.0000009999

    Now that I look at it, if we correctly "round" the 1st one to 0.0000009999 then we would "correctly?" "round" the 2nd one to 0.0000010000.

    I think this im-precision is the nature of the beast when manipulating real numbers.  I don't know what to do about it.

    -Louie

  • llandale
    llandale
    3005 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-06-12T13:54:01Z  
    • llandale
    • ‏2013-04-24T20:48:29Z

    The "real" type has a lot of precision, but when you print it only 6 decimal digits are displayed.  Some time ago I posted a "fStringOfReal()" function which clumsily handled it; other folks posted more elegant solutions.

    -Louie

    I misspoke here.  This has nothing to do with "printing".  Only 6 fractional digits are preserverd when it is converted to a string printing converts it to a string.

  • Adamarla
    Adamarla
    75 Posts

    Re: real in DXL

    ‏2013-06-13T07:44:06Z  

    I have attached my real formatting library code.

    It provides conversion between real and string (Base 10, decimal or scientific notation) in both directions.

    -Adam

    Attachments

  • llandale
    llandale
    3005 Posts

    Re: real in DXL

    ‏2013-06-14T20:47:23Z  
    • Adamarla
    • ‏2013-06-13T07:44:06Z

    I have attached my real formatting library code.

    It provides conversion between real and string (Base 10, decimal or scientific notation) in both directions.

    -Adam

    Wow, zero can be positive or negative and a real can be positive or negative infinity.

    Ran a couple routine tests of the code like this and confirmed the ToString and FromString functions didn't lose any precision, which is a real good sign.

    #include <Includes\NotYetReleased\Lib-Adamarle_Real_ToString.inc>

     

    • void Test(real r1)
    • {
    •  string s1 = Real_ToString(r1)
    •  real   r2 = Real_FromString(s1)
    •  print (r1 == r2) "\t[" s1 "]\t"
    •  print "[" r1 "]\n"  // Doors convert to string
    • }
    • Test(1.4455555555555555555555555557e63)
    • Test( 2.0^1025.0)
    • Test( 2.0^1024.0)
    • Test( 2.0^1023.9999)
      Test(-2.0^1023.9999)

    OK, so that last one DOES lose something.  I wonder what print result [-1.#IND00] means.  Even if IEEE doesn't recognize it, it appears that DOORS does.

    Didn't really study the code, but have these comments when reading it:

    • Yes, while my comments are too verbose, this is far too lean.  What is a "glyph" for example?
    • You have too many hard-coded arbitrary constants; like "20.0" and "1023", "1022"
    • Too many hard-coded "precise" constants, which I think are not quite precise enough.  "0.3010299956639811" doesn't feel right to me.  Should that not be some calculated Exponenet or Logrithm or something?

    In any event it's an order of magnatude better than what I wrote.

    -Louie

  • Adamarla
    Adamarla
    75 Posts

    Re: real in DXL

    ‏2013-06-17T08:22:05Z  
    • llandale
    • ‏2013-06-14T20:47:23Z

    Wow, zero can be positive or negative and a real can be positive or negative infinity.

    Ran a couple routine tests of the code like this and confirmed the ToString and FromString functions didn't lose any precision, which is a real good sign.

    #include <Includes\NotYetReleased\Lib-Adamarle_Real_ToString.inc>

     

    • void Test(real r1)
    • {
    •  string s1 = Real_ToString(r1)
    •  real   r2 = Real_FromString(s1)
    •  print (r1 == r2) "\t[" s1 "]\t"
    •  print "[" r1 "]\n"  // Doors convert to string
    • }
    • Test(1.4455555555555555555555555557e63)
    • Test( 2.0^1025.0)
    • Test( 2.0^1024.0)
    • Test( 2.0^1023.9999)
      Test(-2.0^1023.9999)

    OK, so that last one DOES lose something.  I wonder what print result [-1.#IND00] means.  Even if IEEE doesn't recognize it, it appears that DOORS does.

    Didn't really study the code, but have these comments when reading it:

    • Yes, while my comments are too verbose, this is far too lean.  What is a "glyph" for example?
    • You have too many hard-coded arbitrary constants; like "20.0" and "1023", "1022"
    • Too many hard-coded "precise" constants, which I think are not quite precise enough.  "0.3010299956639811" doesn't feel right to me.  Should that not be some calculated Exponenet or Logrithm or something?

    In any event it's an order of magnatude better than what I wrote.

    -Louie

    Ah, you managed to get an 'Indeterminate' (Not a Number) value with your last test.

    I have added it to the library.

    Note: It will still fail your  (r1 == r2) test because NAN != NAN (which seems stupid to me!)

    Magic Numbers:

    20.0 : to bit shift by 20 because no bit shift operator in dxl

    1023, 1022 : IEEE specified for Normalise / Denormalize

    0.3010299956639811: approximation of [ln(2) / ln(10)] which constrains the result to prevent it overshooting the precise answer

    -Adam

    Attachments

  • jester76
    jester76
    68 Posts

    Re: real in DXL (truncates) (Is there a way to make real type have 99.000000000)

    ‏2013-07-03T20:13:56Z  
    • llandale
    • ‏2013-06-11T19:25:04Z

    It isn't. 

    • 0.0000009999   is being converted to 0.00000099989999999999998, and then truncated to 0.0000009998.
    • 0.00000099999 is being converted to 0.00000099999000000000012, and then truncated to 0.0000009999

    Now that I look at it, if we correctly "round" the 1st one to 0.0000009999 then we would "correctly?" "round" the 2nd one to 0.0000010000.

    I think this im-precision is the nature of the beast when manipulating real numbers.  I don't know what to do about it.

    -Louie

    Found something weird here. If I supply "0.9999999" to the it rounds it to 1.0000000999. Not sure why. Can you please please look into it.

    Thanks for the help

    Jester

  • jester76
    jester76
    68 Posts

    Re: real in DXL

    ‏2013-07-03T20:15:13Z  
    • Adamarla
    • ‏2013-06-17T08:22:05Z

    Ah, you managed to get an 'Indeterminate' (Not a Number) value with your last test.

    I have added it to the library.

    Note: It will still fail your  (r1 == r2) test because NAN != NAN (which seems stupid to me!)

    Magic Numbers:

    20.0 : to bit shift by 20 because no bit shift operator in dxl

    1023, 1022 : IEEE specified for Normalise / Denormalize

    0.3010299956639811: approximation of [ln(2) / ln(10)] which constrains the result to prevent it overshooting the precise answer

    -Adam

    I will try to test this one as well. What is the diff. between private and public funcations.

    How do you restrict the number of digits after the decimal place.

     

    thanks jester

    Updated on 2013-07-04T02:56:57Z at 2013-07-04T02:56:57Z by jester76
  • Adamarla
    Adamarla
    75 Posts

    Re: real in DXL

    ‏2013-07-04T08:43:01Z  
    • jester76
    • ‏2013-07-03T20:15:13Z

    I will try to test this one as well. What is the diff. between private and public funcations.

    How do you restrict the number of digits after the decimal place.

     

    thanks jester

    I've attached my UnitTests that I used during development.

    There is no real difference between private and public funcations. It is just my naming convention to make it clear which functions are safe to call and which are not. The private ones are just internal helpers for the safe functions.

    It converts the numbers without precision loss (a round trip of [Real -> String -> Real] will result in exactly the same real), so there is no functionality to truncate or round. If you need to do that, do it as a separate step afterwards.

    -Adam

    Attachments

  • Adamarla
    Adamarla
    75 Posts

    Re: real in DXL

    ‏2013-07-04T08:57:56Z  
    • llandale
    • ‏2013-06-14T20:47:23Z

    Wow, zero can be positive or negative and a real can be positive or negative infinity.

    Ran a couple routine tests of the code like this and confirmed the ToString and FromString functions didn't lose any precision, which is a real good sign.

    #include <Includes\NotYetReleased\Lib-Adamarle_Real_ToString.inc>

     

    • void Test(real r1)
    • {
    •  string s1 = Real_ToString(r1)
    •  real   r2 = Real_FromString(s1)
    •  print (r1 == r2) "\t[" s1 "]\t"
    •  print "[" r1 "]\n"  // Doors convert to string
    • }
    • Test(1.4455555555555555555555555557e63)
    • Test( 2.0^1025.0)
    • Test( 2.0^1024.0)
    • Test( 2.0^1023.9999)
      Test(-2.0^1023.9999)

    OK, so that last one DOES lose something.  I wonder what print result [-1.#IND00] means.  Even if IEEE doesn't recognize it, it appears that DOORS does.

    Didn't really study the code, but have these comments when reading it:

    • Yes, while my comments are too verbose, this is far too lean.  What is a "glyph" for example?
    • You have too many hard-coded arbitrary constants; like "20.0" and "1023", "1022"
    • Too many hard-coded "precise" constants, which I think are not quite precise enough.  "0.3010299956639811" doesn't feel right to me.  Should that not be some calculated Exponenet or Logrithm or something?

    In any event it's an order of magnatude better than what I wrote.

    -Louie

    Oh, and I don't calculate 0.3010299956639811, because we need to avoid using reals in the conversion or we just introduce more precision loss. It is what it is to ensure the resulting number is suitable for the remaining algorithm.

     

    I used the term Glyph to mean a character in decimal notation {0123456789}. I didn't call it number, because at that point in the algorithm it is just a sequence of Glyphs which need to be altered and arranged to form the number:

    • Correcting for the wrong Glyphs
    • Truncating Glyphs which add no precision
    • Adding a load of zeros
    • Putting the decimal point in the correct place
    • adding the exponent
    Updated on 2013-07-04T09:08:50Z at 2013-07-04T09:08:50Z by Adamarla
  • jester76
    jester76
    68 Posts

    Re: real in DXL

    ‏2013-10-11T12:40:27Z  
    • jester76
    • ‏2013-07-03T20:15:13Z

    I will try to test this one as well. What is the diff. between private and public funcations.

    How do you restrict the number of digits after the decimal place.

     

    thanks jester

    Adam,

    Here are couple of cases I ran into:

    1> 0.009 / 3 returns wrong results => 0.0029999999999999996

    2> 0.008/3 returns wrong results => 0.0026666666666666664 should be 0.0026666666666666667

    Did you know about it. If so can you post fix for it.

     

  • Adamarla
    Adamarla
    75 Posts

    Re: real in DXL

    ‏2013-10-11T13:13:18Z  
    • jester76
    • ‏2013-10-11T12:40:27Z

    Adam,

    Here are couple of cases I ran into:

    1> 0.009 / 3 returns wrong results => 0.0029999999999999996

    2> 0.008/3 returns wrong results => 0.0026666666666666664 should be 0.0026666666666666667

    Did you know about it. If so can you post fix for it.

     

    Initially, I thought you had found a bug in my library, but no...

    The problem is not in this code: it is correctly reporting the result of the calculation.

    The problem is that calculations with real (floating point) numbers do not always result in exactly the correct answer.

    [NB. Not really a 'problem', it is just the way it is]

    Classic example: (0.1+0.2 == 0.3) is false. Why? 

    Because the result of the calculation is actually 0.30000000000000004. Why?

    Because the numbers are stored as the closest number that can be represented in floating point (real). Not all numbers can be represented in floating point (eg. 0.1).

    This happens in all languages which use floating point - javascript, python etc

     

    real r01 = 0.3
    real r02 = 0.1 + 0.2
    real r03 = 0.30000000000000004
     
    print(r01 "\t" Real_ToString(r01) "\n")
    print(r02 "\t" Real_ToString(r02) "\n")
    print(r03 "\t" Real_ToString(r03) "\n")
     
    print(r01 == r02)
    print(r02 == r03)
    print(r01 == r03)
    real r11 = 0.003
    real r12 = 0.009 / 3.0
    real r13 = 0.0029999999999999996
     
    print(r11 "\t" Real_ToString(r11) "\n")
    print(r12 "\t" Real_ToString(r12) "\n")
    print(r13 "\t" Real_ToString(r13) "\n")
     
    print(r11 == r12)
    print(r12 == r13)
    print(r11 == r13)
     
    real r21 = 0.002667
    real r22 = 0.008/3.0
    real r23 = 0.0026666666666666664
     
    print(r21 "\t" Real_ToString(r21) "\n")
    print(r22 "\t" Real_ToString(r22) "\n")
    print(r23 "\t" Real_ToString(r23) "\n")
     
    print(r21 == r22)
    print(r22 == r23)
    print(r21 == r23)

    So why does the DOORS string seem correct... It is rounding it to six decimal places.

    -Adam

    Updated on 2013-10-11T13:34:25Z at 2013-10-11T13:34:25Z by Adamarla
  • llandale
    llandale
    3005 Posts

    Re: real in DXL

    ‏2013-10-11T15:34:36Z  
    • Adamarla
    • ‏2013-10-11T13:13:18Z

    Initially, I thought you had found a bug in my library, but no...

    The problem is not in this code: it is correctly reporting the result of the calculation.

    The problem is that calculations with real (floating point) numbers do not always result in exactly the correct answer.

    [NB. Not really a 'problem', it is just the way it is]

    Classic example: (0.1+0.2 == 0.3) is false. Why? 

    Because the result of the calculation is actually 0.30000000000000004. Why?

    Because the numbers are stored as the closest number that can be represented in floating point (real). Not all numbers can be represented in floating point (eg. 0.1).

    This happens in all languages which use floating point - javascript, python etc

     

    real r01 = 0.3
    real r02 = 0.1 + 0.2
    real r03 = 0.30000000000000004
     
    print(r01 "\t" Real_ToString(r01) "\n")
    print(r02 "\t" Real_ToString(r02) "\n")
    print(r03 "\t" Real_ToString(r03) "\n")
     
    print(r01 == r02)
    print(r02 == r03)
    print(r01 == r03)
    real r11 = 0.003
    real r12 = 0.009 / 3.0
    real r13 = 0.0029999999999999996
     
    print(r11 "\t" Real_ToString(r11) "\n")
    print(r12 "\t" Real_ToString(r12) "\n")
    print(r13 "\t" Real_ToString(r13) "\n")
     
    print(r11 == r12)
    print(r12 == r13)
    print(r11 == r13)
     
    real r21 = 0.002667
    real r22 = 0.008/3.0
    real r23 = 0.0026666666666666664
     
    print(r21 "\t" Real_ToString(r21) "\n")
    print(r22 "\t" Real_ToString(r22) "\n")
    print(r23 "\t" Real_ToString(r23) "\n")
     
    print(r21 == r22)
    print(r22 == r23)
    print(r21 == r23)

    So why does the DOORS string seem correct... It is rounding it to six decimal places.

    -Adam

    I am not "real" good with "real"s.  I wonder if you could do real comparisons

    • (0.1+0.2 == 0.3)

    By converting to a string, "rounding" to perhaps 15 characters, convertinb back to real and THEN comparing?  or perhaps multiplying by 2**15, rounding to whole numbers, and comparing them?  I suppose we'd have to write some "rounding" function.

    -Louie

  • Adamarla
    Adamarla
    75 Posts

    Re: real in DXL

    ‏2013-10-11T16:17:59Z  
    • llandale
    • ‏2013-10-11T15:34:36Z

    I am not "real" good with "real"s.  I wonder if you could do real comparisons

    • (0.1+0.2 == 0.3)

    By converting to a string, "rounding" to perhaps 15 characters, convertinb back to real and THEN comparing?  or perhaps multiplying by 2**15, rounding to whole numbers, and comparing them?  I suppose we'd have to write some "rounding" function.

    -Louie

    Just accept that you cannot represent an infinate number of values with a finite number of bits. Most values have to be stored as a slightly different number. It is just the way it works.

    The normal way to compare floating point numbers is to check they are within some tolarance:

    if( | r1 - r2 | < 0.0000001 ) { ... }

    Most of the complexity in the library is due to the inherant precision loss of floating point calculations, trying to avoid it and to recover from it when unavoidable. That is why I don't calculate the logarithm.

    - Adam

  • EHcnck
    EHcnck
    82 Posts

    Re: real in DXL

    ‏2014-01-21T05:33:08Z  
    • llandale
    • ‏2013-10-11T15:34:36Z

    I am not "real" good with "real"s.  I wonder if you could do real comparisons

    • (0.1+0.2 == 0.3)

    By converting to a string, "rounding" to perhaps 15 characters, convertinb back to real and THEN comparing?  or perhaps multiplying by 2**15, rounding to whole numbers, and comparing them?  I suppose we'd have to write some "rounding" function.

    -Louie

    when comparing numbers(real/double/float) you should only use <, > never ==, <=, >=