[pypy-dev] pickle numpy array from pypy to cpython?

[David Brochart]

Hi,

I verified that this version of PyPy can load a Numpy array that was
pickled by CPython (and do stuff with it), but it looks like a Numpy array
pickled by PyPy cannot be loaded by CPython, because PyPy still uses
'_numpypy.multiarray'
in the pickle string for dumping:
ImportError: No module named _numpypy.multiarray

David.

On Sat, Jul 16, 2016 at 12:07 PM, Matti Picus <matti.picus at gmail.com> wrote:

> The issue with '_numpypy.multiarray' in the pickle string rather than
> 'numpy.core.multiarray' should be fixed on the numpypy_pickle_compat branch
> (thanks to Eli)
> A linux 64 build is available
> http://buildbot.pypy.org/nightly/numpypy_pickle_compat/pypy-c-jit-85727-6d909c810029-linux64.tar.bz2
> .
> Eli or David or anyone who uses numpy pickle, could you check that this
> works as advertised? I am concerned about how compatible our pickling is
> with upstream numpy, but do not really use that feature of numpy so another
> pair of eyes would be nice before merging to default.
>
> Note this requires that http://bitbucket.org/pypy/numpy be installed
> since the Unpickler must be able to import numpy.core.multiarray
> Matti
>
> On 15/07/16 10:47, David Brochart wrote:
>
>> Hi,
>>
>> I'd like to use the (numerical) performances of PyPy as an equivalent to
>> Numba's @jit decorator (https://github.com/davidbrochart/piopio). The
>> only thing preventing that right now is the passing around (pickling) of
>> Numpy arrays, so it would be great to have that compatibility.
>>
>> David.
>>
>> On Mon, Jul 11, 2016 at 6:43 PM, Eli Stevens (Gmail) <
>> wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>
>>     FWVLIW, I think that conforming to upstream numpy makes the most
>>     sense.
>>
>>     I think that the issue would go away if the `_numpypy` module were
>>     renamed to `numpy` (and appropriate things moved into `numpy.core`).
>>     Is there a technical reason to keep the actual implementation in a
>>     separately named module?
>>
>>     Thinking larger picture, would it be possible and sensible to switch
>>     to using the slow cpyext numpy approach for compatability, then
>>     overlay custom implementation of things on top of that when speed is
>>     needed?  I'm imagining a vague inverse of the cpython approach, where
>>     modules are implemented in C when the python performance isn't
>>     acceptable.
>>
>>     Eli
>>
>>     On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org
>>     <mailto:arigo at tunes.org>> wrote:
>>     > Hi Eli, hi Matti,
>>     >
>>     > On 29 June 2016 at 21:37, Eli Stevens (Gmail)
>>     <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>     >> To make sure I'm understanding, are you saying that
>>     upstream/cpython
>>     >> numpy should pick up an alternate way to import multiarray (via
>>     >> _numpypy.multiarray, instead of numpy.core.multiarray)
>>     >
>>     > Hum, in my opinion we should always pickle/unpickle arrays by
>>     > reproducing and expecting the exact same format as CPython's numpy,
>>     > with no warnings.  Any difference (e.g. with complicated dtypes)
>>     is a
>>     > bug that should eventually be fixed.
>>     >
>>     >
>>     > A bientôt,
>>     >
>>     > Armin.
>>     _______________________________________________
>>     pypy-dev mailing list
>>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>>     https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160716/da1ef18a/attachment-0001.html>