Decimal arithmetic, was Re: Python GUI app to impress the boss?

[Chris Gonnerman]

----- Original Message ----- 
From: "Paul Rubin" <phr-n2002b at NOSPAMnightsong.com>


> I'm told that a certain rounding method might not always 
> produce the same answer as doing it the "right" way.  So I 
> asked for a deterministic algorithm for finding the "right" 
> answer in order to compare the results. So far I haven't 
> gotten that algorithm.

Ah.

It's not (really) the rounding algorithm that's at fault.

0.70 * 0.05 in decimal is 0.035.  (Hopefully this, at least,
can be treated as a fact)  The nearest approximation to that 
number which floats can handle (on my current Intel hardware) 
is 0.034999...

>>> 0.7 * 0.05
0.034999999999999996

but if you just put in the number, you get this:

>>> 0.035
0.035000000000000003

so:

>>> 0.035 == (0.7 * 0.05)
0

Gah.  It doesn't matter what "method" or "rules" of
rounding you use, when you can't rely on the basic math
to be right.

You have mentioned the three-for-a-buck bananas many
times.  In decimal floating point, 1/3 = 0.3333...  You
have indicated several times that this result is 
problematic.  Sure, it is.  But, my business-major
friends and customers (intersecting sets) are expecting
that, and in some cases have even alerted me to it
before I thought of it.

None of them would ever expect that the 5% of 70 cents 
calculation would come up with anything other than 3.5
cents, rounded to 4 cents (of course).

Chris Gonnerman -- chris.gonnerman at newcenturycomputers.net
http://newcenturycomputers.net