pickle: huge memory consumption *during* pickling

[Hans Georg Krauthaeuser]

Dear all,

I have a long running application (electromagnetic compatibility 
measurements in mode-stirred chambers over GPIB) that use pickle 
(cPickle) to autosave a class instance with all the measured data from 
time to time.

At the beginning, pickling is quite fast but when the data becomes more 
and more pickling slows down rapidly.

Today morning we reached the situation that it took 6 hours to pickle 
the class instance. The pickle file was than approx. 92 MB (this is ok). 
During pickling the memory consuption of the python proccess was up to 
450 MB (512 MB RAM -> machine was swapping all the time).

My class use data types taken from a c++ class via swig. Don't know if 
that is important...

My feeling is that I'm doing something wrong. But my python knowlegde is 
not so deep to see what that is.

Is there an other way to perform an autosave of an class instance? Shelve?

System: python 2.3.4, Win XP, 1.X GHz class PC, 512 MB ram

Best redards
Hans Georg