[Python-Dev] The role of NotImplemented: What is it for and when should it be used?
Guido van Rossum
guido at python.org
Mon Nov 3 18:05:43 CET 2014
Sorry, was too quick. For immutable types __iop__ may not exist and then
the fallback machinery should work normally using NotImplemented. But if
__iop__ exists it can choose not to allow __rop__, because the type would
presumably change. This is probably more predictable. I don't even know if
the byte code interpreter looks for Not implemented from __iop__.
On Nov 3, 2014 9:00 AM, "Guido van Rossum" <guido at python.org> wrote:
> Not those.
> On Nov 3, 2014 8:56 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
>> On Mon, 3 Nov 2014 08:48:07 -0800
>> Guido van Rossum <guido at python.org> wrote:
>> > Gotta be brief, but NotImplemented is for all binary ops.
>>
>> Even in-place ops?
>>
>> Regards
>>
>> Antoine.
>>
>>
>> > Power may be an
>> > exception because it's ternary?
>> > On Nov 3, 2014 8:08 AM, "Brett Cannon" <brett at python.org> wrote:
>> >
>> > >
>> > >
>> > > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman <ethan at stoneleaf.us>
>> wrote:
>> > >
>> > >> Just to be clear, this is about NotImplemented, not
>> NotImplementedError.
>> > >>
>> > >> tl;dr When a binary operation fails, should an exception be raised
>> or
>> > >> NotImplemented returned?
>> > >>
>> > >
>> > > The docs for NotImplemented suggest it's only for rich comparison
>> methods
>> > > and not all binary operators:
>> > > https://docs.python.org/3/library/constants.html#NotImplemented . But
>> > > then had I not read that I would have said all binary operator methods
>> > > should return NotImplemented when the types are incompatible.
>> > >
>> > > -Brett
>> > >
>> > >
>> > >>
>> > >>
>> > >> When a binary operation in Python is attempted, there are two
>> > >> possibilities:
>> > >>
>> > >> - it can work
>> > >> - it can't work
>> > >>
>> > >> The main reason [1] that it can't work is that the two operands are
>> of
>> > >> different types, and the first type does not know
>> > >> how to deal with the second type.
>> > >>
>> > >> The question then becomes: how does the first type tell Python that
>> it
>> > >> cannot perform the requested operation? The most
>> > >> obvious answer is to raise an exception, and TypeError is a good
>> > >> candidate. The problem with the exception raising
>> > >> approach is that once an exception is raised, Python doesn't try
>> anything
>> > >> else to make the operation work.
>> > >>
>> > >> What's wrong with that? Well, the second type might know how to
>> perform
>> > >> the operation, and in fact that is why we have
>> > >> the reflected special methods, such as __radd__ and __rmod__ -- but
>> if
>> > >> the first type raises an exception the __rxxx__
>> > >> methods will not be tried.
>> > >>
>> > >> Okay, how can the first type tell Python that it cannot do what is
>> > >> requested, but to go ahead and check with the second
>> > >> type to see if it does? That is where NotImplemented comes in -- if
>> a
>> > >> special method (and only a special method)
>> > >> returns NotImplemented then Python will check to see if there is
>> anything
>> > >> else it can do to make the operation succeed;
>> > >> if all attempts return NotImplemented, then Python itself will raise
>> an
>> > >> appropriate exception [2].
>> > >>
>> > >> In an effort to see how often NotImplemented is currently being
>> returned
>> > >> I crafted a test script [3] to test the types
>> > >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict,
>> > >> deque, and OrderedDict with the operations for
>> > >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__,
>> > >> __ilshift__, __imod__, __imul__, __ior__, __ipow__,
>> > >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__,
>> > >> __mul__, __or__, __pow__, __rshift__, __sub__,
>> > >> __truediv__, and __xor__.
>> > >>
>> > >> Here are the results of the 275 tests:
>> > >> ------------------------------------------------------------
>> > >> --------------------
>> > >> testing control...
>> > >>
>> > >> ipow -- Exception <unsupported operand type(s) for ** or pow():
>> 'Control'
>> > >> and 'subtype'> raised
>> > >> errors in Control -- misunderstanding or bug?
>> > >>
>> > >> testing types against a foreign class
>> > >>
>> > >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no
>> attribute
>> > >> 'items'> raised instead of TypeError
>> > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised
>> > >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute
>> > >> 'items'> raised instead of TypeError
>> > >> isub(Counter()) -- Exception <'SomeOtherClass' object has no
>> attribute
>> > >> 'items'> raised instead of TypeError
>> > >>
>> > >>
>> > >> testing types against a subclass
>> > >>
>> > >> mod(str()) -- NotImplemented not returned, TypeError not raised
>> > >>
>> > >> iadd(Counter()) -- Exception <'subtype' object has no attribute
>> 'items'>
>> > >> raised (should have worked)
>> > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised
>> > >> ior(Counter()) -- Exception <'subtype' object has no attribute
>> 'items'>
>> > >> raised (should have worked)
>> > >> isub(Counter()) -- Exception <'subtype' object has no attribute
>> 'items'>
>> > >> raised (should have worked)
>> > >> ------------------------------------------------------------
>> > >> --------------------
>> > >>
>> > >> Two observations:
>> > >>
>> > >> - __ipow__ doesn't seem to behave properly in the 3.x line (that
>> error
>> > >> doesn't show up when testing against 2.7)
>> > >>
>> > >> - Counter should be returning NotImplemented instead of raising an
>> > >> AttributeError, for three reasons [4]:
>> > >> - a TypeError is more appropriate
>> > >> - subclasses /cannot/ work with the current implementation
>> > >> - __iand__ is currently a silent failure if the Counter is
>> empty,
>> > >> and the other operand should trigger a failure
>> > >>
>> > >> Back to the main point...
>> > >>
>> > >> So, if my understanding is correct:
>> > >>
>> > >> - NotImplemented is used to signal Python that the requested
>> operation
>> > >> could not be performed
>> > >> - it should be used by the binary special methods to signal type
>> > >> mismatch failure, so any subclass gets a chance to work.
>> > >>
>> > >> Is my understanding correct? Is this already in the docs somewhere,
>> and
>> > >> I just missed it?
>> > >>
>> > >> --
>> > >> ~Ethan~
>> > >>
>> > >> [1] at least, it's the main reason in my code
>> > >> [2] usually a TypeError, stating either that the operation is not
>> > >> supported, or the types are unorderable
>> > >> [3] test script at the end
>> > >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was
>> > >> rejected]
>> > >>
>> > >> -- 8< ------------------------------------------------------------
>> > >> ----------------
>> > >> from collections import Counter, defaultdict, deque, OrderedDict
>> > >> from fractions import Fraction
>> > >> from decimal import Decimal
>> > >> from enum import Enum
>> > >> import operator
>> > >> import sys
>> > >>
>> > >> py_ver = sys.version_info[:2]
>> > >>
>> > >> types = (
>> > >> bytes, bytearray, str, dict, list, tuple,
>> > >> Enum, Counter, defaultdict, deque, OrderedDict,
>> > >> )
>> > >> numeric_types = int, float, Decimal, Fraction
>> > >>
>> > >> operators = (
>> > >> '__add__', '__and__', '__floordiv__',
>> > >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__',
>> > >> '__imod__', '__imul__', '__ior__', '__ipow__',
>> > >> '__irshift__', '__isub__', '__itruediv__', '__ixor__',
>> > >> '__lshift__', '__mod__', '__mul__',
>> > >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__',
>> > >> '__xor__',
>> > >> )
>> > >>
>> > >> if py_ver >= (3, 0):
>> > >> operators += ('__gt__', '__ge__', '__le__','__lt__')
>> > >>
>> > >> ordered_reflections = {
>> > >> '__le__': '__ge__',
>> > >> '__lt__': '__gt__',
>> > >> '__ge__': '__le__',
>> > >> '__gt__': '__lt__',
>> > >> }
>> > >>
>> > >>
>> > >> # helpers
>> > >>
>> > >> class SomeOtherClass:
>> > >> """"
>> > >> used to test behavior when a different type is passed in to the
>> > >> special methods
>> > >> """
>> > >> def __repr__(self):
>> > >> return 'SomeOtherClass'
>> > >> some_other_class = SomeOtherClass()
>> > >>
>> > >> class MainClassHandled(Exception):
>> > >> """
>> > >> called by base class if both operands are of type base class
>> > >> """
>> > >>
>> > >> class SubClassCalled(Exception):
>> > >> """
>> > >> called by reflected operations for testing
>> > >> """
>> > >>
>> > >> def create_control(test_op):
>> > >> def _any(self, other):
>> > >> if not type(other) is self.__class__:
>> > >> return NotImplemented
>> > >> raise MainClassHandled
>> > >> class Control:
>> > >> "returns NotImplemented when other object is not supported"
>> > >> _any.__name__ = op
>> > >> setattr(Control, test_op, _any)
>> > >> return Control()
>> > >>
>> > >> def create_subtype(test_op, base_class=object):
>> > >> def _any(*a):
>> > >> global subclass_called
>> > >> subclass_called = True
>> > >> raise SubClassCalled
>> > >> class subtype(base_class):
>> > >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ =
>> _any
>> > >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ =
>> _any
>> > >> __and__ = __xor__ = __or__ = _any
>> > >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ =
>> __rfloordiv__ =
>> > >> _any
>> > >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ =
>> __rrshift__ =
>> > >> _any
>> > >> __rand__ = __rxor__ = __ror__ = _any
>> > >> __le__ = __lt__ = __gt__ = __ge__ = _any
>> > >> if issubclass(subtype, (bytes, bytearray)):
>> > >> value = b'hello'
>> > >> elif issubclass(subtype, str):
>> > >> value = 'goodbye'
>> > >> elif issubclass(subtype, (list, tuple)):
>> > >> value = (1, 2, 3)
>> > >> elif issubclass(subtype, (int, float, Decimal, Fraction)):
>> > >> value = 42
>> > >> else:
>> > >> # ignore value
>> > >> return subtype()
>> > >> return subtype(value)
>> > >>
>> > >>
>> > >> # test exceptions
>> > >>
>> > >> # control against some other class
>> > >> print('testing control...\n')
>> > >> errors = False
>> > >> for op in operators:
>> > >> control = create_control(op)
>> > >> op = getattr(operator, op)
>> > >> try:
>> > >> op(control, some_other_class)
>> > >> except TypeError:
>> > >> # the end result of no method existing, or each method
>> called
>> > >> returning
>> > >> # NotImplemented because it does not know how to perform the
>> > >> requested
>> > >> # operation between the two types
>> > >> pass
>> > >> except Exception as exc:
>> > >> errors = True
>> > >> print('%s(%s()) -- Exception <%s> raised instead of
>> TypeError' %
>> > >> (op.__name__, test_type.__name__, exc))
>> > >> else:
>> > >> errors = True
>> > >> print('Control -- TypeError not raised for op %r' % op)
>> > >> if errors:
>> > >> print('errors in Control -- misunderstanding or bug?\n')
>> > >>
>> > >> # control against a subclass
>> > >> errors = False
>> > >> for op in operators:
>> > >> subclass_called = False
>> > >> control = create_control(op)
>> > >> subtype = create_subtype(op, control.__class__)
>> > >> op = getattr(operator, op)
>> > >> try:
>> > >> op(control, subtype)
>> > >> except SubClassCalled:
>> > >> # if the control class properly signals that it doesn't
>> know how
>> > >> to
>> > >> # perform the operation, of if Python notices that a
>> reflected
>> > >> # operation exists, we get here (which is good)
>> > >> pass
>> > >> except MainClassHandled:
>> > >> errors = True
>> > >> print('Control did not yield to subclass for op %r' % op)
>> > >> except Exception as exc:
>> > >> if subclass_called:
>> > >> # exception was subverted to something more appropriate
>> (like
>> > >> # unorderable types)
>> > >> pass
>> > >> errors = True
>> > >> print('%s -- Exception <%s> raised' %
>> > >> (op.__name__, exc))
>> > >> else:
>> > >> errors = True
>> > >> print('Control -- op %r appears to have succeeded (it
>> should not
>> > >> have)' % op)
>> > >> if errors:
>> > >> print('errors in Control -- misunderstanding or bug?\n')
>> > >>
>> > >>
>> > >> # tests
>> > >> print('testing types against a foreign class\n')
>> > >> for test_type in types + numeric_types:
>> > >> errors = False
>> > >> for op in operators:
>> > >> op = getattr(operator, op)
>> > >> try:
>> > >> op(test_type(), some_other_class)
>> > >> except TypeError:
>> > >> pass
>> > >> except Exception as exc:
>> > >> errors = True
>> > >> print('%s(%s()) -- Exception <%s> raised instead of
>> > >> TypeError' %
>> > >> (op.__name__, test_type.__name__, exc))
>> > >> else:
>> > >> print('%s(%s()) -- NotImplemented not returned,
>> TypeError
>> > >> not raised' %
>> > >> (op.__name__, test_type.__name__))
>> > >> if errors:
>> > >> print()
>> > >>
>> > >> print()
>> > >>
>> > >> # test subclasses
>> > >> print('testing types against a subclass\n')
>> > >> for test_type in types:
>> > >> errors = False
>> > >> for op in operators:
>> > >> subclass_called = False
>> > >> if not test_type.__dict__.get(op):
>> > >> continue
>> > >> subclass = create_subtype(op, test_type)
>> > >> op = getattr(operator, op)
>> > >> try:
>> > >> if test_type is str:
>> > >> op('%s', subtype)
>> > >> else:
>> > >> op(test_type(), subtype)
>> > >> except SubClassCalled:
>> > >> # expected, ignore
>> > >> pass
>> > >> except Exception as exc:
>> > >> if subclass_called:
>> > >> # exception raised by subclass was changed
>> > >> pass
>> > >> errors = True
>> > >> print('%s(%s()) -- Exception <%s> raised (should have
>> > >> worked)' %
>> > >> (op.__name__, test_type.__name__, exc))
>> > >> else:
>> > >> errors = True
>> > >> print('%s(%s()) -- NotImplemented not returned,
>> TypeError
>> > >> not raised' %
>> > >> (op.__name__, test_type.__name__))
>> > >> if errors:
>> > >> print()
>> > >> for test_type in numeric_types:
>> > >> errors = False
>> > >> for op in operators:
>> > >> subclass_called = False
>> > >> if not test_type.__dict__.get(op):
>> > >> continue
>> > >> subtype = create_subtype(op, test_type)
>> > >> op = getattr(operator, op)
>> > >> try:
>> > >> op(test_type(), subtype)
>> > >> except SubClassCalled:
>> > >> # expected, ignore
>> > >> pass
>> > >> except Exception as exc:
>> > >> if subclass_called:
>> > >> # exception raised by subclass was changed
>> > >> pass
>> > >> errors = True
>> > >> print('%s(%s()) -- Exception <%s> raised (should have
>> > >> worked)' %
>> > >> (op.__name__, test_type.__name__, exc))
>> > >> else:
>> > >> errors = True
>> > >> print('%s(%s)) -- NotImplemented not returned' %
>> > >> (op.__name__, test_type.__name__))
>> > >> if errors:
>> > >> print()
>> > >> -- 8< ------------------------------------------------------------
>> > >> ----------------
>> > >> _______________________________________________
>> > >> Python-Dev mailing list
>> > >> Python-Dev at python.org
>> > >> https://mail.python.org/mailman/listinfo/python-dev
>> > >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> > >> brett%40python.org
>> > >>
>> > >
>> > > _______________________________________________
>> > > Python-Dev mailing list
>> > > Python-Dev at python.org
>> > > https://mail.python.org/mailman/listinfo/python-dev
>> > > Unsubscribe:
>> > > https://mail.python.org/mailman/options/python-dev/guido%40python.org
>> > >
>> > >
>> >
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141103/e44f41fe/attachment-0001.html>
More information about the Python-Dev
mailing list