Floating point subtraction rounding error (NOT display error)

Nikos Vergas nikos at vergas.gr
Fri Dec 14 04:57:32 EST 2007


> Solved: used round(number,12) in this case for all of the operands of
> my arcsines.  Not pretty, but at least VIM made it easy...

You might have the same problem though:

>>> round(1.000340100000325235000000235,13)
1.0003401000003
>>> round(1.000340100000325235000000235,12)
1.0003401000000001



More information about the Python-list mailing list