pickle->zlib->shelve

[Gordon McMillan]

Thomas Weholt writes:

> I was thinking maybe I could use pickle and compress the pickled
> object too, before storing it into the database and save some
> space. Pickled objects also seem to have lots of repetitive data.
> 
> Now my question is :
> 
> 1. has this been tested and found ineffective in relation to a)
> speed of decompression or b) size of objects not big enough to
> gain any significant size advantage by compression 2. if not
> tested, why? Maybe a subclass of pickle could implement
> compression and decompression of objects?
> 
> Since these scripts are to be run on 400Mhz computers and the
> number of items will not exceed 20-30, objects being relativly
> small, how will the speed be? Any clues??

I regularly use zlib on marshalled data. Works dandy, but for 
small objects (say 100 bytes or so), the zlib'ed version will 
actually be larger. The more data you compress in one shot, 
the better the compression ratio.

- Gordon