[Cython] Auto-pickle progress?

[Robert Bradshaw]

On Tue, Sep 25, 2012 at 5:41 AM, Ian Bell <ian.h.bell at gmail.com> wrote:
> I agree that some of the more powerful elements of Cython would be/are
> difficult or impossible to pickle in a general way.  I have a counter
> use-case where auto-pickling would be very useful.
>
> I have cdef classes that have many (think 50-100) members of double and long
> types, as well as other cdef classes that are also only using
> Cython-friendly data types.  I have implemented pickling for these classes
> and it sucks.  Trying to maintain the list of parameters that need to be
> pickled is a major hassle.  The biggest problem is that if you add add a
> parameter to the class but forget to add it to the pickling function, you
> end up with the parameter not getting reloaded on unpickling.  This is not
> good!  Yes it is up the programmer to do this right, but I could use some
> help.
>
> What I have ended up doing for my other classes is define a __cdict__
> function that returns a dictionary with the values and parameters as defined
> in the cdef class.  In this way, pickling and unpickling is a little bit
> more pleasant.  But it would be fantastic if the __cdict__ could be baked
> into the class automatically.  It would be fine for me if only the things
> that could be easily handled would be returned, and a second list of
> attributes that require more work to be serialized so that the user/coder
> could be sure that all the attributes are being handled and none get
> forgotten.

Exactly. Even if we handled only cdef classes with only (transitively)
picklable fields, this would go a long way. The only question to have
is how to handle changes in the class struct between pickling and
unpickling.

Eventually we could come up with syntax/mechanisms for manually
specifying serialization of more complicated members like double*, but
that's the kind of class you might have to write a __reduce__ method
on anyways.

> Also, the docs on pickling of cdef classes is very sparse, a simple example
> would be very beneficial.  The is one in the examples in the source, but it
> is not easy to find

Agreed.

> On Tue, Sep 25, 2012 at 6:44 AM, Sturla Molden <sturla at molden.no> wrote:
>>
>> On 20.09.2012 21:11, Ian Bell wrote:
>>
>>> Auto-pickling would be tremendously helpful as pickling and unpickling
>>> is one of the most annoying features of working with threads and
>>> processes in python.
>>
>>
>> How should Cython interfere how to pickle a C pointer?
>>
>> cdef class foobar:
>>   cdef double *data
>>
>> A C object can be anything. Cython does not know anything about size,
>> offset or strides, or even if it's safe to take a copy.
>>
>> Example: How to pickle a shared memory buffer? Surely we cannot take a
>> copy, because that would defeat the purpose of "shared" memory. And even if
>> could take a copy, how many bytes should be copied? Do you think an
>> autopickler could have figured this out?
>>
>>
>> https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_sysv.pyx
>>
>>
>> https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_win.pyx
>>
>> On yes, the code is different on Unix and Windows, something the
>> auto-pickler could not possibly know either.
>>
>> Auto-pickling cdef classes is not doable, IMHO.
>>
>>
>> And by the way, implementing a __reduce__ method manually is not very
>> difficult either.
>>
>>
>> Sturla Molden
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>