[Python-ideas] Binary arithmetic does not always call subclasses first

[Stephan Hoyer]

I recently filed this as a bug, and was asked to repost to python-dev or
python-ideas for greater visibility:
http://bugs.python.org/issue30140

Without further ado, here is my original report:

---------------------------
We are writing a system for overloading NumPy operations (see PR [1] and
design doc [2]) that is designed to copy and extend Python's system for
overloading binary operators.

The reference documentation on binary arithmetic [3] states:

> Note: If the right operand's type is a subclass of the left operand’s
type and that subclass provides the reflected method for the operation,
this method will be called before the left operand’s non-reflected method.
This behavior allows subclasses to override their ancestors’ operations.

However, this isn't actually done if the right operand merely inherits from
the left operand's type. In practice, CPython requires that the right
operand defines a different method before it defers to it. Note that the
behavior is different for comparisons, which defer to subclasses regardless
of whether they implement a new method [4].

I think this behavior is a mistake and should be corrected. It is just as
useful to write generic binary arithmetic methods that are well defined on
subclasses as generic comparison operations. In fact, this is exactly the
design pattern we propose for objects implementing special operators like
NumPy arrays (see NDArrayOperatorsMixin in [1] and [2]).

Here is a simple example, of a well-behaved type that implements addition
by wrapping its value and that returns NotImplemented when the other
operand has the wrong type:

class A:
   def __init__(self, value):
       self.value = value
   def __add__(self, other):
       if not isinstance(other, A):
           return NotImplemented
       return type(self)(self.value + other.value)
   __radd__ = __add__
   def __repr__(self):
       return f'{type(self).__name__}({self.value!r})'

class B(A):
    pass

class C(A):
   def __add__(self, other):
       if not isinstance(other, A):
           return NotImplemented
       return type(self)(self.value + other.value)
   __radd__ = __add__

A does not defer to subclass B:

>>> A(1) + B(1)
A(2)

But it does defer to subclass C, which defines new methods (literally
copied/pasted) for __add__/__radd__:

>>> A(1) + C(1)
C(2)

With the current behavior, special operator implementations need to
explicitly account for the possibility that they are being called from a
subclass by returning NotImplemented. My guess is that this is rarely done,
which means that most of these methods are broken when used with
subclasses, or subclasses needlessly reimplement these methods.

Can we fix this logic for Python 3.7?

[1] https://github.com/numpy/numpy/pull/8247
[2]
https://github.com/charris/numpy/blob/406bbc652424fff332f49b0d2f2e5aedd8191d33/doc/neps/ufunc-overrides.rst
[3] https://docs.python.org/3/reference/datamodel.html#object.__ror__
[4] http://bugs.python.org/issue22052

-----------------------------
Mark Dickinson's response:

This is probably worth bringing up on the python-dev or python-ideas
mailing lists for greater visibility. I can't think of any plausible
non-historical reason why it makes sense for comparisons to behave one way
and arithmetic operators another. Altering this might be a PEP-level
change, though.

The "Coercion rules" section[1] of the Python 2.7 docs is a bit more
explicit about the intent:

"""
Exception to the previous item: if the left operand is an instance of a
built-in type or a new-style class, and the right operand is an instance of
a proper subclass of that type or class and overrides the base’s __rop__()
method, the right operand’s __rop__() method is tried before the left
operand’s __op__() method.
"""

so the check for an override was clearly intentional, rather than an
implementation convenience or accident. (It's also clearly intentional in
the source and comments.) The 3.x docs don't have the "and overrides"
language; I haven't figured out why and when that language changed.

[1]
https://docs.python.org/release/2.7.6/reference/datamodel.html#coercion-rules
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170423/5311e43b/attachment.html>