compressed serialization module

[Nick Craig-Wood]

greg <greg at cosc.canterbury.ac.nz> wrote:
>  Mark wrote:
> > Thanks guys.  This is for serializing to disk.  I was hoping to not
> > have to use too many intermediate steps
> 
>  You should be able to use a gzip.GzipFile
>  or bz2.BZ2File and pickle straight into it.

Good idea - that will be much more memory efficient.  Eg

>>> import bz2
>>> import pickle
>>> L = range(100)

>>> f = bz2.BZ2File("z.dat", "wb")
>>> pickle.dump(L, f)
>>> f.close()

>>> f = bz2.BZ2File("z.dat", "rb")
>>> M = pickle.load(f)
>>> f.close()

>>> M == L
True
>>>

(Note that basic pickle protocol is likely to be more compressible
than the binary version!)

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick