those darn exceptions

[Chris Torek]

>On Tue, 21 Jun 2011 01:43:39 +0000, Chris Torek wrote:
>> But how can I know a priori
>> that os.kill() could raise OverflowError in the first place?

In article <4e006912$0$29982$c3e8da3$5496439d at news.astraweb.com>
Steven D'Aprano  <steve+comp.lang.python at pearwood.info> wrote:
>You can't. Even if you studied the source code, you couldn't be sure that 
>it won't change in the future. Or that somebody will monkey-patch 
>os.kill, or a dependency, introducing a new exception.

Indeed.  However, if functions that "know" which exceptions they
themselves can raise "declare" this (through an __exceptions__
attribute for instance), then whoever changes the source or
monkey-patches os.kill can also make the appropriate change to
os.kill.__exceptions__.

>More importantly though, most functions are reliant on their argument. 
>You *cannot* tell what exceptions len(x) will raise, because that depends 
>on what type(x).__len__ does -- and that could be anything. So, in 
>principle, any function could raise any exception.

Yes; this is exactly why you need a type-inference engine to make
this work.  In this case, len() is more (though not quite exactly)
like the following user-defined function:

    def len2(x):
        try:
            fn = x.__len__
        except AttributeError:
            raise TypeError("object of type %r has no len()" % type(x))
        return fn()

eg:

    >>> len(3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: object of type 'int' has no len()
    >>> len2(3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 5, in len2
    TypeError: object of type <type 'int'> has no len()

In this case, len would not have any __exceptions__ field (or if
it does, it would not be a one-element tuple, but I currently think
it makes more sense for many of the built-ins to resort to rules
in the inference engine).  This is also the case for most operators,
e.g., ordinary "+" (or operator.add) is syntactic sugar for:

    first_operand.__add__(second_operand)

or:

    second_operand.__radd__(first_operand)

depending on both operands' types and the first operand's __add__.

The general case is clearly unsolveable (being isomorphic to the
halting problem), but this is not in itself an excuse for attempting
to solve more-specific cases.  A better excuse -- which may well
be "better enough" :-) -- occurs when the specific cases that *can*
be solved are so relatively-rare that the approach degenerates into
uselessness.

It is worth noting that the approach I have in mind does not
survive pickling, which means a very large subset of Python code
is indigestible to a pylint-like exception-inference engine.

>Another question -- is the list of exceptions part of the function's 
>official API? *All* of the exceptions listed, or only some of them?

All the ones directly-raised.  What to do about "invisible"
dependencies (such as those in len2() if len2 is "invisible",
e.g., coded in C rather than Python) is ... less obvious. :-)

>In general, you can't do this at compile-time, only at runtime. There's 
>no point inspecting len.__exceptions__ at compile-time if len is a 
>different function at runtime.

Right.  Which is why pylint is fallible ... yet pylint is still
valuable.  At least, I find it so.  It misses a lot of important
things -- it loses types across list operations, for instance --
but it catches enough to help.  Here is a made-up example based on
actual errors I have found via pylint:

    "doc"
    class Frob(object):
        "doc"
        def __init__(self, arg1, arg2):
            self.arg1 = arg1
            self.arg2 = arg2

        def frob(self, nicate):
            "frobnicate the frob"
            self.arg1 += nicate

        def quux(self):
            "return the frobnicated value"
            example = self # demonstrate that pylint is not using the *name*
            return example.argl # typo, meant arg1
        ...

    $ pylint frob.py
    ************* Module frob
    E1101: 15:Frob.quux: Instance of 'Frob' has no 'argl' member

("Loses types across list operations" means that, e.g.:

        def quux(self):
            return [self][0].argl

hides the type, and hence the typo, from pylint.  At some point I
intend to go in and modify it to track the element-types of list
elements: in "enough" cases, a list's elements all have the same
type, which means we can predict the type of list[i].  If a list
contains mixed types, of course, we have to fall back to the
failure-to-infer case.)

(This also shows that much real code might raise IndexError: any
list subscript that is out of range does so.  So a lot of real
functions *might* raise IndexError, etc., which is another argument
that "in real code, an exception inference engine will wind up
concluding that every line might raise every exception".  Which
might be true, but I still believe, for the moment, that a tool
for inferring exceptions would have some value.)
-- 
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html