[Python-Dev] Python-versus-CPython question for __mul__ dispatch

[Nathaniel Smith]

On Thu, May 14, 2015 at 9:29 PM, Guido van Rossum <guido at python.org> wrote:
> I expect you can make something that behaves like list by defining __mul__
> and __rmul__ and returning NotImplemented.

Hmm, it's fairly tricky, and part of the trick is that you can never
return NotImplemented (because you have to pretty much take over and
entirely replace the normal dispatch rules inside __mul__ and
__rmul__), but see attached for something I think should work.

So I guess this is just how Python's list, tuple, etc. work, and PyPy
and friends need to match...

-n

> On Thursday, May 14, 2015, Stefan Richthofer <Stefan.Richthofer at gmx.de>
> wrote:
>>
>> >>Should I be worried?
>>
>> You mean should *I* be worried ;)
>>
>> Stuff like this is highly relevant for JyNI, so thanks very much for
>> clarifying this
>> subtle behavior. It went onto my todo-list right now to ensure that JyNI
>> will emulate
>> this behavior as soon as I am done with gc-support. (Hopefully it will be
>> feasible,
>> but I can only tell in half a year or so since there are currently other
>> priorities.)
>> Still, this "essay" potentially will save me a lot of time.
>>
>> So, everybody please feel encouraged to post things like this as they come
>> up. Maybe
>> there could be kind of a pitfalls-page somewhere in the docs collecting
>> these things.
>>
>> Best
>>
>> Stefan
>>
>>
>> > Gesendet: Freitag, 15. Mai 2015 um 02:45 Uhr
>> > Von: "Nathaniel Smith" <njs at pobox.com>
>> > An: "Python Dev" <python-dev at python.org>
>> > Betreff: [Python-Dev] Python-versus-CPython question for __mul__
>> > dispatch
>> >
>> > Hi all,
>> >
>> > While attempting to clean up some of the more squamous aspects of
>> > numpy's operator dispatch code [1][2], I've encountered a situation
>> > where the semantics we want and are using are possible according to
>> > CPython-the-interpreter, but AFAICT ought not to be possible according
>> > to Python-the-language, i.e., it's not clear to me whether it's
>> > possible even in principle to implement an object that works the way
>> > numpy.ndarray does in any other interpreter. Which makes me a bit
>> > nervous, so I wanted to check if there was any ruling on this.
>> >
>> > Specifically, the quirk we are relying on is this: in CPython, if you do
>> >
>> >   [1, 2] * my_object
>> >
>> > then my_object's __rmul__ gets called *before* list.__mul__,
>> > *regardless* of the inheritance relationship between list and
>> > type(my_object). This occurs as a side-effect of the weirdness
>> > involved in having both tp_as_number->nb_multiply and
>> > tp_as_sequence->sq_repeat in the C API -- when evaluating "a * b",
>> > CPython tries a's nb_multiply, then b's nb_multiply, then a's
>> > sq_repeat, then b's sq_repeat. Since list has an sq_repeat but not an
>> > nb_multiply, this means that my_object's nb_multiply gets called
>> > before any list method.
>> >
>> > Here's an example demonstrating how weird this is. list.__mul__ wants
>> > an integer, and by "integer" it means "any object with an __index__
>> > method". So here's a class that list is happy to be multiplied by --
>> > according to the ordinary rules for operator dispatch, in the example
>> > below Indexable.__mul__ and __rmul__ shouldn't even get a look-in:
>> >
>> > In [3]: class Indexable(object):
>> >    ...:     def __index__(self):
>> >    ...:         return 2
>> >    ...:
>> >
>> > In [4]: [1, 2] * Indexable()
>> > Out[4]: [1, 2, 1, 2]
>> >
>> > But, if I add an __rmul__ method, then this actually wins:
>> >
>> > In [6]: class IndexableWithMul(object):
>> >    ...:     def __index__(self):
>> >    ...:         return 2
>> >   ...:     def __mul__(self, other):
>> >    ...:         return "indexable forward mul"
>> >    ...:     def __rmul__(self, other):
>> >    ...:         return "indexable reverse mul"
>> >
>> > In [7]: [1, 2] * IndexableWithMul()
>> > Out[7]: 'indexable reverse mul'
>> >
>> > In [8]: IndexableWithMul() * [1, 2]
>> > Out[8]: 'indexable forward mul'
>> >
>> > NumPy arrays, of course, correctly define both __index__ method (which
>> > raises an array on general arrays but coerces to int for arrays that
>> > contain exactly 1 integer), and also defines an nb_multiply slot which
>> > accepts lists and performs elementwise multiplication:
>> >
>> > In [9]: [1, 2] * np.array(2)
>> > Out[9]: array([2, 4])
>> >
>> > And that's all great! Just what we want. But the only reason this is
>> > possible, AFAICT, is that CPython 'list' is a weird type with
>> > undocumented behaviour that you can't actually define using pure
>> > Python code.
>> >
>> > Should I be worried?
>> >
>> > -n
>> >
>> > [1] https://github.com/numpy/numpy/pull/5864
>> > [2] https://github.com/numpy/numpy/issues/5844
>> >
>> > --
>> > Nathaniel J. Smith -- http://vorpus.org
>> > _______________________________________________
>> > Python-Dev mailing list
>> > Python-Dev at python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> > https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.de
>> >
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (on iPad)



-- 
Nathaniel J. Smith -- http://vorpus.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sequence.py
Type: text/x-python
Size: 2868 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150514/56a62e7e/attachment.py>