[Python-Dev] Should instances really be able to dictate the "existence" of special methods?

[Guido van Rossum]

(I suppose this new thread is a result of some research you did regarding
the thread complaining about callable()?)

On Sun, Apr 19, 2015 at 4:03 PM, Eric Snow <ericsnowcurrently at gmail.com>
wrote:

> _PyObject_LookupSpecial is used in place of obj.__getattribute__ for
> looking up special methods.  (As far as I recall it is not exposed in
> the stdlib, e.g. inspect.getattr_special.)  Correct me if I'm wrong
> (please!), but there are two key reasons:
>
>  * access to special methods in spite of obj.__getattribute__
>  * speed
>

Good question! I don't have an easy pointer to the original discussion, but
I do recall that this was introduced in response to some issues with the
original behavior, which looked up dunder methods on the instance and
relied on the general mechanism for binding it to the instance. I don't
think the reason was to circumvent __getattribute__, but your second bullet
rings true: for every +, -, * etc. there would be a (usually failing)
lookup in the instance dict before searching the class dict and then the
base classes etc. There may also have been some confusion where people
would e.g. assign a function of two arguments to x.__add__ and would be
disappointed to find out it was called with only one argument. I think
there were some folks who wanted to fix this by somehow "binding" such
calls to the instance (since there's no easy way otherwise to get the first
argument) but I thought the use case was sufficiently odd that it was
better to avoid it altogether.

In any case, it's not just an optimization -- it's an intentional (though
obscure) feature.


> While _PyObject_LookupSpecial does not do lookup on obj.__dict__ or
> call obj.__getattr__, it does resolve descriptors.  This is important
> particularly since special methods will nearly always be some kind of
> descriptor.  However, one consequence of this is that instances can
> influence whether or not some capability, as relates to the special
> method, is available.  This is accomplished by the descriptor's
> __get__ raising AttributeError.
>

Well, it's not really the instance that raises AttributeError -- it's the
descriptor, which is a separate class (usually but not always a builtin
class, such as property or classmethod). And the descriptor is "owned" by
the class.


> My question is: was this intentional?  Considering the obscure bugs
> that can result (e.g. where did the AttributeError come from?), it
> seems more likely that it is an oversight of an obscure corner case.
>

I'm not sure what you would do to avoid this. You can't very well declare
that a descriptor's __get__ method must not raise AttributeError. It could
be implemented in Python and it could just hit a bug or something. But
perhaps I'm misunderstanding the situation you're describing?


> If that is the case then it would be nice if we could fix
> _PyObject_LookupSpecial to chain any AttributeError coming from
> descr.__get__ into a RuntimeError.  However, I doubt we could get away
> with that at this point.
>

Yeah, I think that ship has sailed. It also seems to be hardly worth trying
to control "double fault" situations like this. (It's not really a double
fault, but it reeks like it.)

I wonder if maybe you're feeling inspired by PEP 479? But that's really a
much more special case, and I don't really want to start down a whole
cascade of trying to "fix" all cases where an AttributeError could be
raised due to a problem in the user's lookup code.


> Also, while it may be appropriate in general to allow instances to
> dictate the availability of attributes/methods (e.g. through
> __getattribute__, __getattr__, or descriptors), I'm not convinced it
> makes sense for special methods.  We are already explicitly
> disconnecting special methods from instances in
> _PyObject_LookupSpecial (except in the case of descriptors...).
>

I'm still a little bit confused why you consider an error from the
descriptor as "dictated by the instance". I think what you're trying to
describe is that there is a method on the class but trying to bind it to
the instance fails. Well, all sorts of things may fails. (In fact very few
things cannot raise an exception in Python.)


> -eric
>
> p.s. I also find it a bit strange that instances have any say at all
> in which methods (i.e. behavior) are *available*.  Certainly instances
> influence behavior, but I always find their impact on method
> availability to be surprising.  Conceptually for me instances are all
> about state and classes about behavior (driven by state).  However, it
> is very rarely that I run into code that takes advantage of the
> opportunity. :)
>

If I understand what you're trying to say, what you're describing is due to
Python's unification of instance variables and methods into attributes.
It's pretty powerful that if x.foo(args) is a method call, you can also
write this as (x.foo)(args), and you can separate the attribute access even
further from the call and pass x.foo to some other function that is
eventually going to call it. Languages that don't do this have to use a
lambda at that point. Like every design choice each choice has its pluses
and minuses, but this is how it's done in Python, and it's not going to
change.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150419/80306bc1/attachment.html>