[Python-Dev] Should instances really be able to dictate the "existence" of special methods?

[Guido van Rossum]

OK, so I think there isn't anything we can or should do here. Yes, it's
possible that type(x).__add__ succeeds but x.__add__ fails. That's how you
spell descriptor. :-) You could also use a random number generator in
__getattribube__...

On Sun, Apr 19, 2015 at 6:36 PM, Eric Snow <ericsnowcurrently at gmail.com>
wrote:

> On Mon, Apr 20, 2015 at 2:20 AM, Guido van Rossum <guido at python.org>
> wrote:
> > (I suppose this new thread is a result of some research you did regarding
> > the thread complaining about callable()?)
>
> Yep. :)
>
> > On Sun, Apr 19, 2015 at 4:03 PM, Eric Snow <ericsnowcurrently at gmail.com>
> > wrote:
> >>
> >> _PyObject_LookupSpecial is used in place of obj.__getattribute__ for
> >> looking up special methods.  (As far as I recall it is not exposed in
> >> the stdlib, e.g. inspect.getattr_special.)  Correct me if I'm wrong
> >> (please!), but there are two key reasons:
> >>
> >>  * access to special methods in spite of obj.__getattribute__
> >>  * speed
> >
> > Good question! I don't have an easy pointer to the original discussion,
> but
> > I do recall that this was introduced in response to some issues with the
> > original behavior, which looked up dunder methods on the instance and
> relied
> > on the general mechanism for binding it to the instance. I don't think
> the
> > reason was to circumvent __getattribute__, but your second bullet rings
> > true: for every +, -, * etc. there would be a (usually failing) lookup in
> > the instance dict before searching the class dict and then the base
> classes
> > etc. There may also have been some confusion where people would e.g.
> assign
> > a function of two arguments to x.__add__ and would be disappointed to
> find
> > out it was called with only one argument. I think there were some folks
> who
> > wanted to fix this by somehow "binding" such calls to the instance (since
> > there's no easy way otherwise to get the first argument) but I thought
> the
> > use case was sufficiently odd that it was better to avoid it altogether.
> >
> > In any case, it's not just an optimization -- it's an intentional (though
> > obscure) feature.
>
> Thanks for explaining.
>
> >> While _PyObject_LookupSpecial does not do lookup on obj.__dict__ or
> >> call obj.__getattr__, it does resolve descriptors.  This is important
> >> particularly since special methods will nearly always be some kind of
> >> descriptor.  However, one consequence of this is that instances can
> >> influence whether or not some capability, as relates to the special
> >> method, is available.  This is accomplished by the descriptor's
> >> __get__ raising AttributeError.
> >
> > Well, it's not really the instance that raises AttributeError -- it's the
> > descriptor, which is a separate class (usually but not always a builtin
> > class, such as property or classmethod). And the descriptor is "owned" by
> > the class.
>
> Sure.  That's what I meant. :)  The instance can influence what the
> descriptor returns.
>
> >> My question is: was this intentional?  Considering the obscure bugs
> >> that can result (e.g. where did the AttributeError come from?), it
> >> seems more likely that it is an oversight of an obscure corner case.
> >
> > I'm not sure what you would do to avoid this. You can't very well declare
> > that a descriptor's __get__ method must not raise AttributeError. It
> could
> > be implemented in Python and it could just hit a bug or something.
>
> Right.  And such a bug will be misinterpreted and obscured and hard to
> unravel.  I ran into this a while back with pickle (which still does
> lookup for special methods on the instance).  Ultimately it's the same
> old problem of not knowing how to interpret an exception that may have
> bubbled up from some other layer.
>
> Like I said, I don't think there's anything to be done about it either
> way.  I just got the feeling that in the case of special methods, the
> descriptor part of lookup should not expect AttributeError to come out
> of the getter.  So I wanted to see if my intuition was correct even if
> the point is essentially irrelevant. :)  At this point, though, I
> think my intuition wasn't quite right, though I still don't think a
> descriptor's getter is the right place to raise AttributeError.
>
> > But
> > perhaps I'm misunderstanding the situation you're describing?
> >
> >>
> >> If that is the case then it would be nice if we could fix
> >> _PyObject_LookupSpecial to chain any AttributeError coming from
> >> descr.__get__ into a RuntimeError.  However, I doubt we could get away
> >> with that at this point.
> >
> > Yeah, I think that ship has sailed. It also seems to be hardly worth
> trying
> > to control "double fault" situations like this. (It's not really a double
> > fault, but it reeks like it.)
> >
> > I wonder if maybe you're feeling inspired by PEP 479? But that's really a
> > much more special case, and I don't really want to start down a whole
> > cascade of trying to "fix" all cases where an AttributeError could be
> raised
> > due to a problem in the user's lookup code.
>
> Nah.  It isn't about fixing all the cases nor directly related to PEP
> 479.  Instead it is in response to one obscure corner case (the
> behavior of callable).
>
> >> Also, while it may be appropriate in general to allow instances to
> >> dictate the availability of attributes/methods (e.g. through
> >> __getattribute__, __getattr__, or descriptors), I'm not convinced it
> >> makes sense for special methods.  We are already explicitly
> >> disconnecting special methods from instances in
> >> _PyObject_LookupSpecial (except in the case of descriptors...).
> >
> > I'm still a little bit confused why you consider an error from the
> > descriptor as "dictated by the instance".
>
> Sorry, I should have been more clear.  The descriptor part of attr
> lookup involves a way for the instance to influence the outcome of the
> lookup (at the discretion of the descriptor).  I didn't mean to imply
> that the instance has a direct role in special method lookup.
>
> > I think what you're trying to
> > describe is that there is a method on the class but trying to bind it to
> the
> > instance fails. Well, all sorts of things may fails. (In fact very few
> > things cannot raise an exception in Python.)
> >
> >>
> >> -eric
> >>
> >> p.s. I also find it a bit strange that instances have any say at all
> >> in which methods (i.e. behavior) are *available*.  Certainly instances
> >> influence behavior, but I always find their impact on method
> >> availability to be surprising.  Conceptually for me instances are all
> >> about state and classes about behavior (driven by state).  However, it
> >> is very rarely that I run into code that takes advantage of the
> >> opportunity. :)
> >
> >
> > If I understand what you're trying to say, what you're describing is due
> to
> > Python's unification of instance variables and methods into attributes.
> It's
> > pretty powerful that if x.foo(args) is a method call, you can also write
> > this as (x.foo)(args), and you can separate the attribute access even
> > further from the call and pass x.foo to some other function that is
> > eventually going to call it. Languages that don't do this have to use a
> > lambda at that point. Like every design choice each choice has its pluses
> > and minuses, but this is how it's done in Python, and it's not going to
> > change.
>
> Again I wasn't very clear.  Rather than the unification of attributes,
> I'm referring to how descriptors can raise AttributeError in __get__
> and  it gets interpreted as "attribute missing" in
> object.__getattribute__ (and similar lookup methods).  However, just
> in the context of descriptors it makes a little more sense, even if
> still consider it too clever.  I think I was just focusing too much on
> the influence of instances on descriptors. :)
>
> -eric
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150419/4fec3a77/attachment-0001.html>