round issue
Robin Becker
robin at reportlab.com
Mon Jul 12 05:52:00 EDT 2010
A client wants to know why his db number -9.85 gets displayed by some simple
code as -9.8
I looked at the number and see that
>>> -9.85
-9.8499999999999996
ie I expect simple rounding to produce the observed result and indeed
>>> '%.1f' % -9.85
'-9.8'
however, when I use round I get an unexpected result ie
>>> round(-9.85,1)
-9.9000000000000004
according to its definition
> round(x[, n])¶
> Return the floating point value x rounded to n digits after the decimal point.
> If n is omitted, it defaults to zero. The result is a floating point number.
> Values are rounded to the closest multiple of 10 to the power minus n;
> if two multiples are equally close, rounding is done away from 0 (so. for example,
> round(0.5) is 1.0 and round(-0.5) is -1.0).
so looking at the absolute differences I see
>>> abs(-9.9 - -9.85)
0.050000000000000711
>>> abs(-9.8 - -9.85)
0.049999999999998934
ie the -9.8 value appears closer and at least to a primitive test
>>> abs(-9.9 - -9.85) > abs(-9.8 - -9.85)
True
the distance from the -9.9 result is larger, however, that may be because the
model numbers for -9.8 & -9.9 differ in distance from the true 10**-n values eg
>>> -9.9
-9.9000000000000004
>>> -9.8
-9.8000000000000007
What value should round(-9.85,1) return? Is the result explainable in python (ie
without resort to the internal FP representations etc etc)?
--
Robin Becker
More information about the Python-list
mailing list