[Cython] Auto-pickle progress?

[Ian Bell]

I agree that some of the more powerful elements of Cython would be/are
difficult or impossible to pickle in a general way.  I have a counter
use-case where auto-pickling would be very useful.

I have cdef classes that have many (think 50-100) members of double and
long types, as well as other cdef classes that are also only using
Cython-friendly data types.  I have implemented pickling for these classes
and it sucks.  Trying to maintain the list of parameters that need to be
pickled is a major hassle.  The biggest problem is that if you add add a
parameter to the class but forget to add it to the pickling function, you
end up with the parameter not getting reloaded on unpickling.  This is not
good!  Yes it is up the programmer to do this right, but I could use some
help.

What I have ended up doing for my other classes is define a __cdict__
function that returns a dictionary with the values and parameters as
defined in the cdef class.  In this way, pickling and unpickling is a
little bit more pleasant.  But it would be fantastic if the __cdict__ could
be baked into the class automatically.  It would be fine for me if only the
things that could be easily handled would be returned, and a second list of
attributes that require more work to be serialized so that the user/coder
could be sure that all the attributes are being handled and none get
forgotten.

Also, the docs on pickling of cdef classes is very sparse, a simple example
would be very beneficial.  The is one in the examples in the source, but it
is not easy to find

Ian

On Tue, Sep 25, 2012 at 6:44 AM, Sturla Molden <sturla at molden.no> wrote:

> On 20.09.2012 21:11, Ian Bell wrote:
>
>  Auto-pickling would be tremendously helpful as pickling and unpickling
>> is one of the most annoying features of working with threads and
>> processes in python.
>>
>
> How should Cython interfere how to pickle a C pointer?
>
> cdef class foobar:
>   cdef double *data
>
> A C object can be anything. Cython does not know anything about size,
> offset or strides, or even if it's safe to take a copy.
>
> Example: How to pickle a shared memory buffer? Surely we cannot take a
> copy, because that would defeat the purpose of "shared" memory. And even if
> could take a copy, how many bytes should be copied? Do you think an
> autopickler could have figured this out?
>
> https://github.com/**sturlamolden/sharedmem-numpy/**blob/master/sharedmem/
> **sharedmemory_sysv.pyx<https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_sysv.pyx>
>
> https://github.com/**sturlamolden/sharedmem-numpy/**blob/master/sharedmem/
> **sharedmemory_win.pyx<https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_win.pyx>
>
> On yes, the code is different on Unix and Windows, something the
> auto-pickler could not possibly know either.
>
> Auto-pickling cdef classes is not doable, IMHO.
>
>
> And by the way, implementing a __reduce__ method manually is not very
> difficult either.
>
>
> Sturla Molden
>
> ______________________________**_________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/**mailman/listinfo/cython-devel<http://mail.python.org/mailman/listinfo/cython-devel>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20120925/b3087cf8/attachment.html>