Decimal arithmetic, was Re: Python GUI app to impress the boss?

[Dennis Lee Bieber]

Chris Gonnerman fed this fish to the penguins on Tuesday 01 October 
2002 09:46 pm:


> 
>>>> 0.7 * 0.05
> 0.034999999999999996
> 
> but if you just put in the number, you get this:
> 
>>>> 0.035
> 0.035000000000000003
> 
> so:
> 
>>>> 0.035 == (0.7 * 0.05)
> 0

        The method that almost any good book on scientific computation gives 
for comparison is to never test for equality for floating point 
calculations. Rather one tests for an epsilon.

>>> a = .7 * .05
>>> b = .035
>>> print a, b
0.035 0.035
>>> a
0.034999999999999996
>>> b
0.035000000000000003
>>> eps = 0.00005
>>> abs(a-b) < eps
1
>>>
>>> eps = 0.001
>>> abs(a-b) < eps
1


--
 > ============================================================== <
 >   wlfraed at ix.netcom.com  | Wulfraed  Dennis Lee Bieber  KD6MOG <
 >      wulfraed at dm.net     |       Bestiaria Support Staff       <
 > ============================================================== <
 >        Bestiaria Home Page: http://www.beastie.dm.net/         <
 >            Home Page: http://www.dm.net/~wulfraed/             <