[Python-Dev] The role of NotImplemented: What is it for and when should it be used?
Antoine Pitrou
solipsis at pitrou.net
Mon Nov 3 17:55:10 CET 2014
On Mon, 3 Nov 2014 08:48:07 -0800
Guido van Rossum <guido at python.org> wrote:
> Gotta be brief, but NotImplemented is for all binary ops.
Even in-place ops?
Regards
Antoine.
> Power may be an
> exception because it's ternary?
> On Nov 3, 2014 8:08 AM, "Brett Cannon" <brett at python.org> wrote:
>
> >
> >
> > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman <ethan at stoneleaf.us> wrote:
> >
> >> Just to be clear, this is about NotImplemented, not NotImplementedError.
> >>
> >> tl;dr When a binary operation fails, should an exception be raised or
> >> NotImplemented returned?
> >>
> >
> > The docs for NotImplemented suggest it's only for rich comparison methods
> > and not all binary operators:
> > https://docs.python.org/3/library/constants.html#NotImplemented . But
> > then had I not read that I would have said all binary operator methods
> > should return NotImplemented when the types are incompatible.
> >
> > -Brett
> >
> >
> >>
> >>
> >> When a binary operation in Python is attempted, there are two
> >> possibilities:
> >>
> >> - it can work
> >> - it can't work
> >>
> >> The main reason [1] that it can't work is that the two operands are of
> >> different types, and the first type does not know
> >> how to deal with the second type.
> >>
> >> The question then becomes: how does the first type tell Python that it
> >> cannot perform the requested operation? The most
> >> obvious answer is to raise an exception, and TypeError is a good
> >> candidate. The problem with the exception raising
> >> approach is that once an exception is raised, Python doesn't try anything
> >> else to make the operation work.
> >>
> >> What's wrong with that? Well, the second type might know how to perform
> >> the operation, and in fact that is why we have
> >> the reflected special methods, such as __radd__ and __rmod__ -- but if
> >> the first type raises an exception the __rxxx__
> >> methods will not be tried.
> >>
> >> Okay, how can the first type tell Python that it cannot do what is
> >> requested, but to go ahead and check with the second
> >> type to see if it does? That is where NotImplemented comes in -- if a
> >> special method (and only a special method)
> >> returns NotImplemented then Python will check to see if there is anything
> >> else it can do to make the operation succeed;
> >> if all attempts return NotImplemented, then Python itself will raise an
> >> appropriate exception [2].
> >>
> >> In an effort to see how often NotImplemented is currently being returned
> >> I crafted a test script [3] to test the types
> >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict,
> >> deque, and OrderedDict with the operations for
> >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__,
> >> __ilshift__, __imod__, __imul__, __ior__, __ipow__,
> >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__,
> >> __mul__, __or__, __pow__, __rshift__, __sub__,
> >> __truediv__, and __xor__.
> >>
> >> Here are the results of the 275 tests:
> >> ------------------------------------------------------------
> >> --------------------
> >> testing control...
> >>
> >> ipow -- Exception <unsupported operand type(s) for ** or pow(): 'Control'
> >> and 'subtype'> raised
> >> errors in Control -- misunderstanding or bug?
> >>
> >> testing types against a foreign class
> >>
> >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> >> 'items'> raised instead of TypeError
> >> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> >> 'items'> raised instead of TypeError
> >> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute
> >> 'items'> raised instead of TypeError
> >>
> >>
> >> testing types against a subclass
> >>
> >> mod(str()) -- NotImplemented not returned, TypeError not raised
> >>
> >> iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'>
> >> raised (should have worked)
> >> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> >> ior(Counter()) -- Exception <'subtype' object has no attribute 'items'>
> >> raised (should have worked)
> >> isub(Counter()) -- Exception <'subtype' object has no attribute 'items'>
> >> raised (should have worked)
> >> ------------------------------------------------------------
> >> --------------------
> >>
> >> Two observations:
> >>
> >> - __ipow__ doesn't seem to behave properly in the 3.x line (that error
> >> doesn't show up when testing against 2.7)
> >>
> >> - Counter should be returning NotImplemented instead of raising an
> >> AttributeError, for three reasons [4]:
> >> - a TypeError is more appropriate
> >> - subclasses /cannot/ work with the current implementation
> >> - __iand__ is currently a silent failure if the Counter is empty,
> >> and the other operand should trigger a failure
> >>
> >> Back to the main point...
> >>
> >> So, if my understanding is correct:
> >>
> >> - NotImplemented is used to signal Python that the requested operation
> >> could not be performed
> >> - it should be used by the binary special methods to signal type
> >> mismatch failure, so any subclass gets a chance to work.
> >>
> >> Is my understanding correct? Is this already in the docs somewhere, and
> >> I just missed it?
> >>
> >> --
> >> ~Ethan~
> >>
> >> [1] at least, it's the main reason in my code
> >> [2] usually a TypeError, stating either that the operation is not
> >> supported, or the types are unorderable
> >> [3] test script at the end
> >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was
> >> rejected]
> >>
> >> -- 8< ------------------------------------------------------------
> >> ----------------
> >> from collections import Counter, defaultdict, deque, OrderedDict
> >> from fractions import Fraction
> >> from decimal import Decimal
> >> from enum import Enum
> >> import operator
> >> import sys
> >>
> >> py_ver = sys.version_info[:2]
> >>
> >> types = (
> >> bytes, bytearray, str, dict, list, tuple,
> >> Enum, Counter, defaultdict, deque, OrderedDict,
> >> )
> >> numeric_types = int, float, Decimal, Fraction
> >>
> >> operators = (
> >> '__add__', '__and__', '__floordiv__',
> >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__',
> >> '__imod__', '__imul__', '__ior__', '__ipow__',
> >> '__irshift__', '__isub__', '__itruediv__', '__ixor__',
> >> '__lshift__', '__mod__', '__mul__',
> >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__',
> >> '__xor__',
> >> )
> >>
> >> if py_ver >= (3, 0):
> >> operators += ('__gt__', '__ge__', '__le__','__lt__')
> >>
> >> ordered_reflections = {
> >> '__le__': '__ge__',
> >> '__lt__': '__gt__',
> >> '__ge__': '__le__',
> >> '__gt__': '__lt__',
> >> }
> >>
> >>
> >> # helpers
> >>
> >> class SomeOtherClass:
> >> """"
> >> used to test behavior when a different type is passed in to the
> >> special methods
> >> """
> >> def __repr__(self):
> >> return 'SomeOtherClass'
> >> some_other_class = SomeOtherClass()
> >>
> >> class MainClassHandled(Exception):
> >> """
> >> called by base class if both operands are of type base class
> >> """
> >>
> >> class SubClassCalled(Exception):
> >> """
> >> called by reflected operations for testing
> >> """
> >>
> >> def create_control(test_op):
> >> def _any(self, other):
> >> if not type(other) is self.__class__:
> >> return NotImplemented
> >> raise MainClassHandled
> >> class Control:
> >> "returns NotImplemented when other object is not supported"
> >> _any.__name__ = op
> >> setattr(Control, test_op, _any)
> >> return Control()
> >>
> >> def create_subtype(test_op, base_class=object):
> >> def _any(*a):
> >> global subclass_called
> >> subclass_called = True
> >> raise SubClassCalled
> >> class subtype(base_class):
> >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any
> >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any
> >> __and__ = __xor__ = __or__ = _any
> >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ =
> >> _any
> >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ =
> >> _any
> >> __rand__ = __rxor__ = __ror__ = _any
> >> __le__ = __lt__ = __gt__ = __ge__ = _any
> >> if issubclass(subtype, (bytes, bytearray)):
> >> value = b'hello'
> >> elif issubclass(subtype, str):
> >> value = 'goodbye'
> >> elif issubclass(subtype, (list, tuple)):
> >> value = (1, 2, 3)
> >> elif issubclass(subtype, (int, float, Decimal, Fraction)):
> >> value = 42
> >> else:
> >> # ignore value
> >> return subtype()
> >> return subtype(value)
> >>
> >>
> >> # test exceptions
> >>
> >> # control against some other class
> >> print('testing control...\n')
> >> errors = False
> >> for op in operators:
> >> control = create_control(op)
> >> op = getattr(operator, op)
> >> try:
> >> op(control, some_other_class)
> >> except TypeError:
> >> # the end result of no method existing, or each method called
> >> returning
> >> # NotImplemented because it does not know how to perform the
> >> requested
> >> # operation between the two types
> >> pass
> >> except Exception as exc:
> >> errors = True
> >> print('%s(%s()) -- Exception <%s> raised instead of TypeError' %
> >> (op.__name__, test_type.__name__, exc))
> >> else:
> >> errors = True
> >> print('Control -- TypeError not raised for op %r' % op)
> >> if errors:
> >> print('errors in Control -- misunderstanding or bug?\n')
> >>
> >> # control against a subclass
> >> errors = False
> >> for op in operators:
> >> subclass_called = False
> >> control = create_control(op)
> >> subtype = create_subtype(op, control.__class__)
> >> op = getattr(operator, op)
> >> try:
> >> op(control, subtype)
> >> except SubClassCalled:
> >> # if the control class properly signals that it doesn't know how
> >> to
> >> # perform the operation, of if Python notices that a reflected
> >> # operation exists, we get here (which is good)
> >> pass
> >> except MainClassHandled:
> >> errors = True
> >> print('Control did not yield to subclass for op %r' % op)
> >> except Exception as exc:
> >> if subclass_called:
> >> # exception was subverted to something more appropriate (like
> >> # unorderable types)
> >> pass
> >> errors = True
> >> print('%s -- Exception <%s> raised' %
> >> (op.__name__, exc))
> >> else:
> >> errors = True
> >> print('Control -- op %r appears to have succeeded (it should not
> >> have)' % op)
> >> if errors:
> >> print('errors in Control -- misunderstanding or bug?\n')
> >>
> >>
> >> # tests
> >> print('testing types against a foreign class\n')
> >> for test_type in types + numeric_types:
> >> errors = False
> >> for op in operators:
> >> op = getattr(operator, op)
> >> try:
> >> op(test_type(), some_other_class)
> >> except TypeError:
> >> pass
> >> except Exception as exc:
> >> errors = True
> >> print('%s(%s()) -- Exception <%s> raised instead of
> >> TypeError' %
> >> (op.__name__, test_type.__name__, exc))
> >> else:
> >> print('%s(%s()) -- NotImplemented not returned, TypeError
> >> not raised' %
> >> (op.__name__, test_type.__name__))
> >> if errors:
> >> print()
> >>
> >> print()
> >>
> >> # test subclasses
> >> print('testing types against a subclass\n')
> >> for test_type in types:
> >> errors = False
> >> for op in operators:
> >> subclass_called = False
> >> if not test_type.__dict__.get(op):
> >> continue
> >> subclass = create_subtype(op, test_type)
> >> op = getattr(operator, op)
> >> try:
> >> if test_type is str:
> >> op('%s', subtype)
> >> else:
> >> op(test_type(), subtype)
> >> except SubClassCalled:
> >> # expected, ignore
> >> pass
> >> except Exception as exc:
> >> if subclass_called:
> >> # exception raised by subclass was changed
> >> pass
> >> errors = True
> >> print('%s(%s()) -- Exception <%s> raised (should have
> >> worked)' %
> >> (op.__name__, test_type.__name__, exc))
> >> else:
> >> errors = True
> >> print('%s(%s()) -- NotImplemented not returned, TypeError
> >> not raised' %
> >> (op.__name__, test_type.__name__))
> >> if errors:
> >> print()
> >> for test_type in numeric_types:
> >> errors = False
> >> for op in operators:
> >> subclass_called = False
> >> if not test_type.__dict__.get(op):
> >> continue
> >> subtype = create_subtype(op, test_type)
> >> op = getattr(operator, op)
> >> try:
> >> op(test_type(), subtype)
> >> except SubClassCalled:
> >> # expected, ignore
> >> pass
> >> except Exception as exc:
> >> if subclass_called:
> >> # exception raised by subclass was changed
> >> pass
> >> errors = True
> >> print('%s(%s()) -- Exception <%s> raised (should have
> >> worked)' %
> >> (op.__name__, test_type.__name__, exc))
> >> else:
> >> errors = True
> >> print('%s(%s)) -- NotImplemented not returned' %
> >> (op.__name__, test_type.__name__))
> >> if errors:
> >> print()
> >> -- 8< ------------------------------------------------------------
> >> ----------------
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> >> brett%40python.org
> >>
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
>
More information about the Python-Dev
mailing list