Comparing float and decimal

[Tim Roberts]

Mark Dickinson <dickinsm at gmail.com> wrote:

>On Sep 25, 8:55 am, Tim Roberts <t... at probo.com> wrote:
>> Marc 'BlackJack' Rintsch <bj_... at gmx.net> wrote:
>> >0.1 actually is
>>
>> >In [98]: '%.50f' % 0.1
>> >Out[98]: '0.10000000000000000555111512312578270211815834045410'
>> >?
>>
>> ....  0.1 in an IEEE 784 double is this:
>>
>>      0.100000000000000088817841970012523233890533447265625
>
>I get (using Python 2.6):
>
>>>> n, d = 0.1.as_integer_ratio()
>>>> from decimal import Decimal, getcontext
>>>> getcontext().prec = 100
>>>> Decimal(n)/Decimal(d)
>Decimal('0.1000000000000000055511151231257827021181583404541015625')
>
>which is a lot closer to Marc's answer.  Looks like your float
>approximation to 0.1 is 6 ulps out.  :-)

Yes, foolishness on my part.  The hex is 3FB99999_9999999A,
so we're looking at 19999_9999999A / 2^56 or
  7205759403792794
 -------------------
  72057594037927936

which is the number that Marc, Nick, and you all describe.  Apologies all
around.  I actually dropped one 9 the first time around.

Adding one more weird data point, here's what I get trying Marc's original
sample on my Windows box:

   C:\tmp>python
   Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on
win32
   Type "help", "copyright", "credits" or "license" for more information.
    >>> '%.50f' % 0.1
    '0.10000000000000001000000000000000000000000000000000'
    >>> 
I assume this is the Microsoft C run-time library at work.
-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.