Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.
Originally posted by: rabbottThere is a standard problem with computers making floating point calculations in decimal (base 10) because the computer generates and stores them in binary (base 2). Essentially, what you�re looking at is a single decimal representation of what could be one of several approximate binary values. For example:
a = 2.905
b = 2.9 + .005
c = 2.9 + .004999999998
d = 2.9 + .005000000000000001
emit a,b,c,d
emit a.round(-2) as ar
emit b.round(-2) as br
emit c.round(-2) as cr
emit d.round(-2) as dr
Will output:
a:double | b:double | c:double | d:double | ar:double | br:double | cr:double | dr:double
2.905 | 2.905 | 2.905 | 2.905 | 2.91 | 2.91 | 2.9 | 2.91
Note that all four values show as 2.905, but the rounding of all four is not the same.
The best way to deal with this is to convert the calculation to integer math. For example:
round(2.905, -2)
Gives you either 2.91 OR 2.90 randomly
round (2.905, -2, 1)
Gives you 2.91 consistently but does so for any value over 2.90
round (2.905, -2, -1)
Gives you 2.90 consistently for any value
To ensure you get 2.91 for any value of 2.905 through 2.909, you would want to convert the calculation to integer math:
round(int(1000 * 2.905), 1)/1000
Attached is a graph containing the above example and solution.
- Bob
Attachments:
RoundingFloats.brg