pickle: huge memory consumption *during* pickling

[Robert Brewer]

Hans Georg Krauthaeuser wrote:
> I have a long running application (electromagnetic compatibility 
> measurements in mode-stirred chambers over GPIB) that use pickle 
> (cPickle) to autosave a class instance with all the measured 
> data from 
> time to time.
> 
> At the beginning, pickling is quite fast but when the data 
> becomes more 
> and more pickling slows down rapidly.
> 
> Today morning we reached the situation that it took 6 hours to pickle 
> the class instance. The pickle file was than approx. 92 MB 
> (this is ok). 
> During pickling the memory consuption of the python proccess 
> was up to 
> 450 MB (512 MB RAM -> machine was swapping all the time).
> 
> My class use data types taken from a c++ class via swig. 
> Don't know if 
> that is important...
> 
> My feeling is that I'm doing something wrong. But my python 
> knowlegde is 
> not so deep to see what that is.
> 
> Is there an other way to perform an autosave of an class 
> instance? Shelve?

Shelve just uses pickle:

try:
    from cPickle import Pickler, Unpickler
except ImportError:
    from pickle import Pickler, Unpickler

...but it might help if the bottleneck is in your file-handling.


Robert Brewer
MIS
Amor Ministries
fumanchu at amor.org