[Numpy-discussion] Another nice write-up about floating point issues.

[Chris Barker]

Hi all,

I just stumbled upon this paper by William Kahan, a Professor at
Berkeley that is well known for his work on the first Intel math
co-processor, and the development of IEE 754 (plus a lot of other
stuff). I took a course with him at Berkeley, and the man is brilliant.
So brilliant that is is very hard to follow him, as a student, if you
are not so sharp. I'm not, and I just squeeked by and missed a great
deal of what he tried to teach. I did gain a good appreciation of the
complexity of floating point arithmetic, however. Here is the paper as a
PDF:

http://www.cs.berkeley.edu/%7ewkahan/MgtkMath.pdf

There's a lot of other good stuff on his web page at:

http://www.cs.berkeley.edu/%7ewkahan/

THe paper is entitled "Marketing versus Mathematics". In particular,
take a look at the section on Quattro Pro on Page 13, for a nice
discussion of binary arithmetic displayed as decimal, which is an often
brought up subject in the Python newsgroups. He discusses how you'd
really have to display more than the 15 digits that Quattro displayed,
to avoid confusion.

"""
But no such cure can be liberated from little annoyances:
0.8  entered would display as 0.80000000000000004 to 17 sig. dec.;
"""
At the Python prompt:
>>> 0.8
0.80000000000000004

More evidence that the current Python way is "best", if confusing to
newbies. This would be another good paper to point people towards that
ask about floating point on the newsgroups.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov