[Python-Dev] Python-versus-CPython question for __mul__ dispatch

[Nathaniel Smith]

On Sat, May 16, 2015 at 1:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 16 May 2015 at 07:35, Nathaniel Smith <njs at pobox.com> wrote:
>> On Thu, May 14, 2015 at 11:53 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>> On Thu, May 14, 2015 at 9:29 PM, Guido van Rossum <guido at python.org> wrote:
>>>> I expect you can make something that behaves like list by defining __mul__
>>>> and __rmul__ and returning NotImplemented.
>>>
>>> Hmm, it's fairly tricky, and part of the trick is that you can never
>>> return NotImplemented (because you have to pretty much take over and
>>> entirely replace the normal dispatch rules inside __mul__ and
>>> __rmul__), but see attached for something I think should work.
>>>
>>> So I guess this is just how Python's list, tuple, etc. work, and PyPy
>>> and friends need to match...
>>
>> For the record, it looks like PyPy does already have a hack to
>> implement this -- they do it by having a hidden flag on the built-in
>> sequence types which the implementations of '*' and '+' check for, and
>> if it's found it triggers a different rule for dispatching to the
>> __op__ methods:
>>     https://bitbucket.org/pypy/pypy/src/a1a494787f4112e42f50c6583e0fea18db3fb4fa/pypy/objspace/descroperation.py?at=default#cl-692
>
> Oh, that's rather annoying that the PyPy team implemented bug-for-bug
> compatibility there, and didn't follow up on the operand precedence
> bug report to say that they had done so. We also hadn't previously
> been made aware that NumPy is relying on this operand precedence bug
> to implement publicly documented API behaviour, so fixing it *would*
> break end user code :(

I don't think any of us were aware of it either :-).

It is a fairly obscure case -- it only comes up specifically if you
have a single-element integer array that you are trying to multiply by
a list that you expect to be auto-coerced to an array. If Python
semantics were such that this became impossible to handle correctly
then we would survive. (We've certainly survived worse, e.g.
arr[array_of_indices] += 1 silently gives the wrong/unexpected result
when array_of_indices has duplicate entries, and this bites people
constantly. Unfortunately I can't see any reasonable way to fix this
within Python's semantics, so... oh well.)

But yeah, given that we're at a point where list dispatch actually has
worked this way forever and across multiple interpreter
implementations, I think it's de facto going to end up part of the
language specification unless someone does something pretty quick...

> I guess that means someone in the numeric community will need to write
> a PEP to make this "try the other operand first" "feature" part of the
> language specification, so that other interpreters can implement it up
> front, rather than all having to come up with their own independent
> custom hacks just to make NumPy work.

I'll make a note...

> P.S. It would also be nice if someone could take on the PEP for a
> Python level buffer API for 3.6: http://bugs.python.org/issue13797

At a guess, if you want to find people who have this itch strong
enough to try scratching it, then probably numpy users are actually
not your best bet, b/c if you have numpy then you already have
workarounds. In particular, numpy still supports a legacy Python level
buffer export API:

   http://docs.scipy.org/doc/numpy/reference/arrays.interface.html#python-side

So if all you want is to hand a buffer to numpy (rather than to an
arbitrary PEP 3118 consumer) then this works fine, and if you do need
an arbitrary PEP 3118 consumer then you can use numpy as an adaptor
(use __array_interface__ to convert your object to ndarray -> ndarray
supports the PEP 3118 API).

-n

-- 
Nathaniel J. Smith -- http://vorpus.org