[pypy-dev] pickle numpy array from pypy to cpython?

[Eli Stevens (Gmail)]

I am not surprised that my current branch doesn't cover all cases; it
was specifically targeted at my exact, singular use case.

I'll work on making something more general, as well as improving test coverage.

On Sat, Jul 16, 2016 at 9:29 AM, Matti Picus <matti.picus at gmail.com> wrote:
> So it seems the tests are lacking. Someone should:
> - go through all the existing calls to dumps in tests and add "assert
> '_numpypy' not in data"
> - add tests for scalars
> - fix so the tests pass
> Matti
>
>
> On 16/07/16 07:40, David Brochart wrote:
>
> To be more precise, PyPy pickling of Numpy arrays works fine, it is when
> PyPy pickles a Numpy scalar that I get the error.
> David.
>
> On Sat, Jul 16, 2016 at 2:04 PM, David Brochart <david.brochart at gmail.com>
> wrote:
>>
>> Hi,
>>
>> I verified that this version of PyPy can load a Numpy array that was
>> pickled by CPython (and do stuff with it), but it looks like a Numpy array
>> pickled by PyPy cannot be loaded by CPython, because PyPy still uses
>> '_numpypy.multiarray' in the pickle string for dumping:
>> ImportError: No module named _numpypy.multiarray
>>
>> David.
>>
>> On Sat, Jul 16, 2016 at 12:07 PM, Matti Picus <matti.picus at gmail.com>
>> wrote:
>>>
>>> The issue with '_numpypy.multiarray' in the pickle string rather than
>>> 'numpy.core.multiarray' should be fixed on the numpypy_pickle_compat branch
>>> (thanks to Eli)
>>> A linux 64 build is available
>>> http://buildbot.pypy.org/nightly/numpypy_pickle_compat/pypy-c-jit-85727-6d909c810029-linux64.tar.bz2.
>>> Eli or David or anyone who uses numpy pickle, could you check that this
>>> works as advertised? I am concerned about how compatible our pickling is
>>> with upstream numpy, but do not really use that feature of numpy so another
>>> pair of eyes would be nice before merging to default.
>>>
>>> Note this requires that http://bitbucket.org/pypy/numpy be installed
>>> since the Unpickler must be able to import numpy.core.multiarray
>>> Matti
>>>
>>> On 15/07/16 10:47, David Brochart wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'd like to use the (numerical) performances of PyPy as an equivalent to
>>>> Numba's @jit decorator (https://github.com/davidbrochart/piopio). The only
>>>> thing preventing that right now is the passing around (pickling) of Numpy
>>>> arrays, so it would be great to have that compatibility.
>>>>
>>>> David.
>>>>
>>>> On Mon, Jul 11, 2016 at 6:43 PM, Eli Stevens (Gmail)
>>>> <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>>>
>>>>     FWVLIW, I think that conforming to upstream numpy makes the most
>>>>     sense.
>>>>
>>>>     I think that the issue would go away if the `_numpypy` module were
>>>>     renamed to `numpy` (and appropriate things moved into `numpy.core`).
>>>>     Is there a technical reason to keep the actual implementation in a
>>>>     separately named module?
>>>>
>>>>     Thinking larger picture, would it be possible and sensible to switch
>>>>     to using the slow cpyext numpy approach for compatability, then
>>>>     overlay custom implementation of things on top of that when speed is
>>>>     needed?  I'm imagining a vague inverse of the cpython approach,
>>>> where
>>>>     modules are implemented in C when the python performance isn't
>>>>     acceptable.
>>>>
>>>>     Eli
>>>>
>>>>     On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org
>>>>     <mailto:arigo at tunes.org>> wrote:
>>>>     > Hi Eli, hi Matti,
>>>>     >
>>>>     > On 29 June 2016 at 21:37, Eli Stevens (Gmail)
>>>>     <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>>>     >> To make sure I'm understanding, are you saying that
>>>>     upstream/cpython
>>>>     >> numpy should pick up an alternate way to import multiarray (via
>>>>     >> _numpypy.multiarray, instead of numpy.core.multiarray)
>>>>     >
>>>>     > Hum, in my opinion we should always pickle/unpickle arrays by
>>>>     > reproducing and expecting the exact same format as CPython's
>>>> numpy,
>>>>     > with no warnings.  Any difference (e.g. with complicated dtypes)
>>>>     is a
>>>>     > bug that should eventually be fixed.
>>>>     >
>>>>     >
>>>>     > A bientôt,
>>>>     >
>>>>     > Armin.
>>>>     _______________________________________________
>>>>     pypy-dev mailing list
>>>>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>>>>     https://mail.python.org/mailman/listinfo/pypy-dev
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pypy-dev mailing list
>>>> pypy-dev at python.org
>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>>
>>
>
>