[Python-Dev] Pickling w/ low overhead

[David Ascher]

On Tue, 3 Aug 1999, M.-A. Lemburg wrote:

> Hmm, types can register their own pickling/unpickling functions
> via copy_reg, so they can access the self.write method in pickle.py
> to implement the write to file interface. 

Are you sure?  My understanding of copy_reg is, as stated in the doc:

pickle (type, function[, constructor]) 
    Declares that function should be used as a ``reduction'' function for
    objects of type or class type. function should return either a string
    or a tuple. The optional constructor parameter, if provided, is a
    callable object which can be used to reconstruct the object when
    called with the tuple of arguments returned by function at pickling
    time.  

How does one access the 'self.write method in pickle.py'?

> Perhaps some lazy pickling wrapper would help fix this in general:
> an object which calls back into the to-be-pickled object to
> access the data rather than store the data in a huge string.

Right.  That's an idea.

> Yet another idea would be using memory mapped files instead
> of strings as temporary storage (but this is probably hard to implement
> right and not as portable).

That's a very interesting idea!  I'll try that -- it might just be the
easiest way to do this.  I think that portability isn't a huge concern --
the folks who are coming up with the speed issue are on platforms which
have mmap support.

Thanks for the suggestions.

--david