Decimal arithmatic, was Re: Python GUI app to impress the boss?

[Dennis Lee Bieber]

Chris Gonnerman fed this fish to the penguins on Sunday 29 September 
2002 03:33 pm:


> You ask how it is a different problem.  Simple.  A human,
> training in decimal math in grade school, knows that
> 
>     1/3 = 0.3333333...
> 
> and would *expect* it to round to 0.33 (still in two decimal

        So, in 2 decimal place arithmetic

        a = 1.00 / 3.00
        b = a * 3.00

gives
        b = 0.99

which is NOT equal to 1... But this is okay because it's done is 
"decimal arithmetic"? <G>

        The telling phrase is "training in DECIMAL math"... Most computers 
don't work in decimal math, so the user /should/ be ready to learn the 
ins&outs of binary floating point...

        For every example of "what is wrong with binary floating point" 
someone else can find a similar case when using decimal math. Even VB's 
"currency" type (which looks to be a scaled 64-bit integer) carries 4 
decimal places -- which allows for "proper" results when displaying 
final results in two decimal places with rounding.

        1/3 => 0.3333   internally
        3*0.3333 => 0.9999
        1.00 for display after rounding.


--
 > ============================================================== <
 >   wlfraed at ix.netcom.com  | Wulfraed  Dennis Lee Bieber  KD6MOG <
 >      wulfraed at dm.net     |       Bestiaria Support Staff       <
 > ============================================================== <
 >        Bestiaria Home Page: http://www.beastie.dm.net/         <
 >            Home Page: http://www.dm.net/~wulfraed/             <