Possible bug (was Re: numpy, overflow, inf, ieee, and rich comparison)

[Huaiyu Zhu]

On Wed, 11 Oct 2000 22:44:20 -0400, Tim Peters <tim_one at email.msn.com> wrote:
>
>Python was not designed with IEEE-754 in mind, and *anything* that happens
>wrt Infs, NaNs, and all other 754 features in Python is purely an accident
>that can and does vary wildly across platforms.  And, as you've discovered
>in this case, can vary wildly also across even a single library, depending
>on config options.  We cannot consider this to be a bug since Python has had
>no *intended* behavior whatsoever wrt 754 gimmicks.  We can and do consider
>gripes about these accidents to be feature requests.

Yes it varies widely in Python.  But isn't that precisely because Python
does not support IEEE?  If Python does not actively undermine IEEE on
platforms that have it, would it still vary widely across these platforms?

>[Guido]
>> Incidentally, math.exp(800) returns inf in 1.5, and raises
>> OverflowError in 2.0.  So at least it's consistent.
>
>[Huaiyu]
>> That is not consistent at all.

Oops, I misread.  I thought it was meant that inf and OverflowError were
consistent with each other. 

>[Guido
>> 1.5.2 links with -lieee while 2.0 doesn't.  Removing -lieee from the
>> 1.5.2 link line makes is raise OverflowError too.  Adding it to the
>> 2.0 link line makes it return 0.0 for exp(-1000) and inf for
>> exp(1000).
>
>[Huaiyu]
>> If using ieee is as simple as setting such a flag, there is no
>> reason at all not to use it.
>
>The patch to stop setting -lieee was contributed by a Python user who
>claimed it fixed bugs on *their* platform.  That's "the reason".  We don't
>want to screw them either.

Q: What bug?  Which platform?  Maybe there is another way that has less
   impact on others?  How to find out?
   
>I understand that 754 semantics can be invaluable.  So does Guido.  There's
>no argument about that.  But Python doesn't yet support them, and wishing it
>did doesn't make it so.  If linking with -lieee happens to give you the
>semantics you want on your platform today, you're welcome to build with that
>switch.  It appears to be a bad choice for *most* Python users, though (more
>below), so we don't want to do it in the std 2.0 distro.

If there is a way to fix the bug on that other platform while still keep
-lieee on Linux, would that be acceptable?


> 754
>is a Very Good Thing, but is Evil in subset form (see Kahan's (justified!)
>vilification of Java on this point);

Interesting! I've glanced through some of Prof Kahan's writings.  It appears
that he favors raising flags that can be checked optionally, and he is
against mandatary raising exceptions.  Some quotes (Warning: highly
selective quoting.  See http://http.cs.berkeley.edu/~wkahan/ for details):

  Designers of operating systems tend to incorporate all trap-handling into
  their handiwork, thereby burdening floating-point exception-handling with
  unnecessary and intolerable overheads. ...

  A common mistake is to treat exceptions as errors ...
  Exceptions that reveal errors are merely messengers.  What turns an
  exception into an error is bad exception-handling.

  Attempts to cope decently with all exceptions inevitably run into
  unresolved dilemmas sooner or later ...

In the paper deriding Java, he wrote

  Java linguistically confuses the issues about floating-point Exceptions:
  Java, like C++, misuses the word "Exception" to mean what IEEE 754 calls a
  "Trap."  Java has no words for the five float-point Events that IEEE 754
  calls "Exceptions": 

  Invalid Operation, Overflow, Division-by-Zero, Underflow, Inexact Result
  
     These events are _not errors_ unless they are handled badly.

  They are called "Exceptions" because to any policy for handling them,
  imposed in advance upon all programmers by the computer system, some
  programmers will have good reason to take exception.
  ...

  IEEE 754 specifies a default policy for each of these kinds of
  floating-point exception:

  1. Singal the event by raising an appropriate one of the five flags, if it
  has not already be raised.

  2. (Pre)substitute a default value for what would have been the result of
  the exception operation:
    [ A table describing when to set NaN, Inf, etc ]

  3. Resume execution of the program as if nothing exceptional had occured.

  With these default values, IEEE 754's floating-point becomes an
  Algebraically Completed system; this means the computer's every algebraic
  operation produces a well-defined result for all operands.

>From these quotes and others I understand that he wants f.p. exceptions to
be merely messanges, not errors that force programmers to take extra
actions, unless the programmer choose to.  So what aspect of Java did he
think is deficient?  It is the "Trap" style exception that mandate a users
action.

He told "A cautionary Tale of the Ariane 5", the European rocket that
mulfunctioned after lift off.  It turned out that a piece of software
produced an unimportant exception that is not caught by a handler and caused
debugging data to be dumped to a critical memory location.  This would have
been avoided if the exception merely raised a flag and returned a NaN, and
the program continued.

So do we want every Python program to bomb whenever there is a floating
point value outside bound and is not trapped by try/except?

>Your biggest obstacle in getting Python support for 754 will in fact be
>opposition from number-crunchers.  I've been slinging fp for 21 years, 15 of
>those writing compilers and math libraries for "supercomputer" vendors. 
...
> ironically, 754 is hardest to sell to
>those who could benefit from it the most.

So far the only concern from number-crunchers voiced here is that it might
allow sqrt(-1)==0, which is not the case.  I was a fortran programmer
before.  I switched to using IEEE ten years ago as soon as it was available
on the machine I was using.  So there are different opinions.

>> ...
>> The language should never take over or subvert decisions based on
>> numerical analysis.
>
>Which is why a subset of 754 is evil.  754 requires that the user be able to
>*choose* whether or not, e.g., overflow signals an exception.  Your crusade
>to insist that it never raise an exception is as least as bad (I think it's
>much worse) as Python's most common accidental behavior (where overflow from
>libm usually raises an exception).  One size does not fit all.

Obviously we agree on the benefit of having a choice.  But we have different
perspective on what choice means.  Your want a choice to force user to take
explicit action, I want a choice to ignore these events.  The ideal solution
might be to have a sys.fp_exception class.  Each exception would set a flag
like fp_exception.overflow so that user can check them.  There could also be
a variable like fp_exception.force_trap, which when set, could make each
f.p. exception to produce a Python exception.  This would confirm to IEEE
754 and give both of us a choice we want.

Now that we are in feature freeze for 2.0, we can't get both choices.  Can
we adopt the one that does not break behaviors on previous official release,
even if that's only by accident?

>[Tim]
>> Ignoring ERANGE entirely is not at all the same behavior as 1.5.2, and
>> current code certainly relies on detecting overflows in math functions.
>
>[Huaiyu]
>> As Guido observed ERANGE is not generated with ieee, even for
>> overflow.  So it is the same behavior as 1.5.2.
>
>You've got a bit of a case of tunnel vision here, Huaiyu.  Yes, in the
>specific case of gcc+glibc+Linux, ignoring ERANGE returned from exp() is
>what 1.5.2 acted like.  But you have not proposed to ignore it merely from
>ERANGE returned from exp() in the specific case of gcc+glibc+Linux.  This
>code runs on many dozens of platforms, and, e.g., as I suggested before, it
>looks like HPUX has always set errno on both overflow and underflow.  MS
>Windows sets it on overflow but not underflow.  Etc.  We have to worry about
>the effects on all platforms.

So now I know why people reported that NaN and Inf were broken on HPUX and
Windows and some others.  If Python had simply ignored the flags ...

>
>> Besides, no correct numerical code should depend on exceptions like
>> this unless the machine is incapable of handling Inf and NaN.
>
>Nonsense.  754 provides the option to raise an exception on overflow, or
>not, at the user's discretion, precisely because exceptions are sometimes
>more appropriate than Infs of NaNs.  Kahan himself isn't happy with Infs and
>NaNs because they're too often too gross a clue (see his writings on
>"presubstitution" for a more useful approach).

>From Kahan's writings I'm tempted to say that 754 allowed signalling mainly
for political reasons.  Yes Infs and NaNs are often too gross.  But consider
a 100x100 array with a few Infs here and there, I'd say that's finer control
than simply halting the program flow.   How else could you express the idea
of "mark these as invalid and continue with the rest" efficiently?

Could you please point a URL to "presubstitution"?  In the paragraphs I
quoted the only mention of presubstitute is with NaNs and Inf, etc.

>I personally doubt that, but realize it may be true.  That's why he have
>beta releases.  So far yours is the only feedback we've heard (thanks!), and
>as a result we're going to change 2.0 to stop griping about underflow, and
>do so in a way that will actually work across all platforms.  We're probably
>going to break some HPUX programs as a result; but they were relying on
>accidents too.

Why change behavior in the last minute?  If we find out why HPUX users
submitted the patch, we might find a way to remove the bug for them without
affecting any other platform.

>Python does not support 754 today.  Period.  I would like it to, but not in
>any half-assed, still platform-dependent, still random crap-shoot, still
>random subset of 754 features, still rigidly inflexible, way.  When it does
>support it, Guido & I will argue that it should enable (what 754 calls) the
>overflow, divide-by-0 and invalid operation exceptions by default, and
>disable the underflow and inexact exceptions by default.  This ensures that,
>under the default, an infinity or NaN can never be created from
>non-exceptional inputs without triggering an exception. Not only is that
>best for newbies, you'll find it's the *only* scheme for a default that can
>be sold to working number-crunchers (been there, done that, at Kendall
>Square Research).  It also matches Guido's original, pre-754, *intent* for
>how Python numerics should work by default (it is, e.g., no accident that
>Python has always had an OverflowError exception but never an UnderflowError
>one).
>
>And that corresponds to the change Guido says <wink> I'm going to make in
>mathmodule.c:  suppress complaints about underflow, but let complaints about
>overflow go through.  This is not a solution, it's a step on a path towards
>a solution.  The next step (which will not happen for 2.0!) is to provide an
>explicit way to, from Python, disable overflow checking, and that's simply
>part of providing the control and inquiry functions mandated by 754.  Then
>code that would rather deal with Infs than exceptions can, at its explicit
>discretion.

You are not confirming to 754 until you allow all the exceptions to go
through with well defined values as default.  Raising Python exceptions on
some of them does not confirm to 754, as the default IEEE 754 behavior on
f.p. exception is not a Python exception, but just a raised flag.

Before that happens, what is the motive to make IEEE less usable for those
who have it (pretty much every one nowadays), even if just by accident?

I would classify raising exceptions as a very rigidly inflexible way. Maybe
I missed something.  If I'm dealing with an array of 1000 elements, is there
a way to ignore a few elements and continue with the rest?

>Apparently Python on libc+glibc+Linux never raised OverflowError from math.*
>functions in 1.5.2 (although it did on most other platforms I know about).

Is it just glibc or is it glibc+Linux?  Here's test on my machine

Without -lieee
  x       exp(x)     errno   errmessage
-745.0  4.94066e-324    0    Success
-746.0             0   34  Numerical result out of range
 709.0  8.21841e+307   34  Numerical result out of range
 710.0           inf   34  Numerical result out of range
sqrt( -1.0) =   nan,   33  Numerical argument out of domain
    
And with -lieee
  x       exp(x)     errno   errmessage
-745.0  4.94066e-324    0    Success
-746.0             0    0    Success
 709.0  8.21841e+307    0    Success
 710.0           inf    0    Success
sqrt( -1.0) =   nan,    0    Success
	

On all the other platforms, what happens when you use ieee confirming flags?
I'd bet they behave pretty much the same.  Anyone on other platform care to
check out?  Here's the C program

#include <stdio.h>
#include <math.h>
#include <errno.h>

int main () {
  double x, y;
  printf("  x       exp(x)     errno   errmessage\n");
  for (x=-745; x>-747; x--)
  {
    y = exp(x);
    printf("%6.1f %13.6g  %3d  %9s\n", x, y, errno, strerror(errno));
  }
  for (x=709; x<711; x++)
  {
    y = exp(x);
    printf("%6.1f %13.6g  %3d  %9s\n", x, y, errno, strerror(errno));
  }
		
  x = -1;
  y = sqrt(x);
  printf("sqrt(%5.1f) = %5.3g,  %3d  %9s\n", x, y, errno, strerror(errno));
  return 0;
}
      
The above two results are produced with the following
CFLAGS = -lm
CFLAGS = -lm -lieee

>
>> Likewise sqrt(-1.) needs to stop, not put a zero and keep going.
>
>Nobody has proposed changing anything about libm domain (as opposed to
>range) errors (although Huaiyu probably should if he's serious about his
>flavor of 754 subsetism -- I have no idea what gcc+glibc+Linux did here on
>1.5.2, but it *should* have silently returned a NaN (not a zero) without
>setting errno if it was *self*-consistent -- anyone care to check that
>under -lieee?:

My subsetism?  I didn't know I have one.  I'm all for having a place to hold
those global flags, if that's what you mean missing from the subset. I'm
only against excluding the subset of default 754 behavior.

But just to clarify some wordings you quoted in another post: I meant that
sqrt(-1)==0 is insane, whether silent or not.  Raising an exception is sane,
although OverflowError is not so reasonable.  Raising a ValueError is
reasonable.  But just returning NaN without a fuss is even better.  My
favorite is returning 1j, as MatPy does.

>
>just-another-day-of-universal-python-harmony-ly y'rs  - tim

if-it-ain't-broke-don't-fix-it-ly y'rs 

Huaiyu