Unpickling crashing my machine

[Skip Montanaro]

    Peter> Pierre-Frédéric Caillaud wrote:

    >> way #1 :
    >> for object in objects :
    >>     cPickle.dump( object, myfile, -1 )
    >> 
    >> way #2 :
    >> p = cPickle.Pickler( myfile, -1 )
    >> for object in objects :
    >>     p.dump( object )
    >> 
    >> Loading the file generated by #2 :
    >> - the progress counter runs as fast as #1
    >> - eats all memory, then swap
    ...

    Peter> I believe that what you see is the excessive growth of that
    Peter> cache.

Correct.  If each object dumped is independent of all the other objects
being dumped you should clear the memo after each dump() call in the second
case:

    p = cPickle.Pickler( myfile, -1 )
    for object in objects :
        p.dump( object )
	p.memo_clear()

The first case doesn't suffer from this problem because each call to
cPickle.dump() creates a new Pickler, and thus a new memo.

Skip