1.090516455488E9 / 1000000.000 ???

[Fredrik Lundh]

"Math" wrote:

> I got a simple and probably stupid newby question..
> When I compute:
> 1.090516455488E9 / 1000000
> the result is 1090516455.49
> Should be 1090516455.488

assuming you meant ~1090, it is:

>>> 1.090516455488E9 / 1000000
1090.516455488

unless you round it off to 12 significant digits, which is
what str() does:

>>> str(1.090516455488E9 / 1000000)
'1090.51645549'

on the other hand, if you meant E15 instead, you get
another problem:

>>> 1.090516455488E15 / 1000000
1090516455.4879999 # oops!

>>> str(1.090516455488E15 / 1000000)
'1090516455.49'

the reason is of course that floating point are stored as a limited
number of base 2-digits internally.  what you get when you con-
vert that back to decimal depends on the resulting number itself,
not on the number of significant digits you used on the original
expression.

for more on this topic, see

    http://docs.python.org/tut/node16.html

and consider using

    http://www.python.org/doc/lib/module-decimal.html

instead.

> I know something with float and //...

if you know how floats work, how come you're surprised by
rounding issues ?

</F>