[Python-Dev] Unpickling memory usage problem, and a proposed solution

[Dan Gindikin]

Collin Winter <collinwinter <at> google.com> writes:
> I should add that, adding the necessary bookkeeping to remove only
> unused PUTs (instead of the current all-or-nothing scheme) should not
> be hard. I'd watch out for a further performance/memory hit; the
> pickling benchmarks in the benchmark suite should help assess this.
> The current optimization penalizes pickling to speed up unpickling,
> which made sense when optimizing pickles that would go into memcache
> and be read out 13-15x more often than they were written.

This wouldn't help our use case, your code needs the entire pickle
stream to be in memory, which in our case would be about 475mb, this
is on top of the 300mb+ data structures that generated the pickle
stream.