cPickle asymptotic performance?

[Eric Jonas]

On Thu, 2008-06-12 at 20:57 +0200, Hrvoje Niksic wrote:
> Eric Jonas <jonas at MIT.EDU> writes:
> 
> > I've done some benchmarking while attempting to serialize my (large)
> > graph data structure with cPickle; I'm seeing superlinear performance
> > (plotting it seems to suggest n^2 where n is the number of nodes of my
> > graph), in the duration of the pickle.dump calls and I can't quite
> > figure out why.
> 
> Try gc.disable() before loading the pickle, and gc.enable() after.
> 
> > Is cPickle's behavior known to be O(n^2)?
> 
> No, but the garbage collector's sometimes is.


Wow, that totally fixed it -- we went from 1200 seconds to 60 seconds.
I'm somewhat shocked -- is the occasionally-abysmal behavior of the GC
documented anywhere? 
			...Eric