[issue45876] Improve accuracy of stdev functions in statistics

Mark Dickinson report at bugs.python.org
Fri Nov 26 16:59:02 EST 2021


Mark Dickinson <dickinsm at gmail.com> added the comment:

Concrete example of int/int not being correctly rounded on systems using x87 instructions: on those systems, I'd expect to see 1/2731 return a result of 0.00036616623947272064 (0x1.7ff4005ffd002p-12), as a result of first rounding to 64-bit precision and then to 53-bit. The correctly-rounded result is 0.0003661662394727206 (0x1.7ff4005ffd001p-12).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45876>
_______________________________________


More information about the Python-bugs-list mailing list