[Python-Dev] Pickling w/ low overhead

[M.-A. Lemburg]

David Ascher wrote:
> 
> An issue which has dogged the NumPy project is that there is (to my
> knowledge) no way to pickle very large arrays without creating strings
> which contain all of the data.  This can be a problem given that NumPy
> arrays tend to be very large -- often several megabytes, sometimes much
> bigger.  This slows things down, sometimes a lot, depending on the
> platform. It seems that it should be possible to do something more
> efficient.
> 
> Two alternatives come to mind:
> 
>  -- define a new pickling protocol which passes a file-like object to the
>     instance and have the instance write itself to that file, being as
>     efficient or inefficient as it cares to.  This protocol is used only
>     if the instance/type defines the appropriate slot.  Alternatively,
>     enrich the semantics of the getstate interaction, so that an object
>     can return partial data and tell the pickling mechanism to come back
>     for more.
> 
>  -- make pickling of objects which support the buffer interface use that
>     inteface's notion of segments and use that 'chunk' size to do
>     something more efficient if not necessarily most efficient.  (oh, and
>     make NumPy arrays support the buffer interface =).  This is simple
>     for NumPy arrays since we want to pickle "everything", but may not be
>     what other buffer-supporting objects want.
> 
> Thoughts?  Alternatives?

Hmm, types can register their own pickling/unpickling functions
via copy_reg, so they can access the self.write method in pickle.py
to implement the write to file interface. Don't know how this
would be done for cPickle.c though.

For instances the situation is different since there is no
dispatching done on a per-class basis. I guess an optional argument
could help here.

Perhaps some lazy pickling wrapper would help fix this in general:
an object which calls back into the to-be-pickled object to
access the data rather than store the data in a huge string.

Yet another idea would be using memory mapped files instead
of strings as temporary storage (but this is probably hard to implement
right and not as portable).

Dunno... just some thoughts.

-- 
Marc-Andre Lemburg
______________________________________________________________________
Y2000:                                                   150 days left
Business:                                      http://www.lemburg.com/
Python Pages:                           http://www.lemburg.com/python/