[Numpy-discussion] float128 casting rounding as if it were float64
Matthew Brett
matthew.brett at gmail.com
Tue Oct 11 14:17:47 EDT 2011
Hi,
While struggling with floating point precision, I ran into this:
In [52]: a = 2**54+3
In [53]: a
Out[53]: 18014398509481987L
In [54]: np.float128(a)
Out[54]: 18014398509481988.0
In [55]: np.float128(a)-1
Out[55]: 18014398509481987.0
The line above tells us that float128 can exactly represent 2**54+3,
but the line above that says that np.float128(2**54+3) rounds upwards
as if it were a float64:
In [59]: np.float64(a)
Out[59]: 18014398509481988.0
In [60]: np.float64(a)-1
Out[60]: 18014398509481988.0
Similarly:
In [66]: np.float128('1e308')
Out[66]: 1.000000000000000011e+308
In [67]: np.float128('1e309')
Out[67]: inf
Is it possible that float64 is being used somewhere in float128 casting?
Best,
Matthew
More information about the NumPy-Discussion
mailing list