performance of pickling & large lists :-(

[Andreas Leitgeb]

Being new in Python, I try lots of things, whether actually
useful or not :-)

I had a really large list (of about 4 million entries) of lists of 
 each 10 int's.  The process had a footprint of about 300MB then.

Then I tried to pickle it to a file (not to string :-).  I'd have 
expected, that while the file would become quite large, hardly any 
additional memory would be needed, as the pickler would iterate through
the elements, create a reversible string-representation for each
and write it out, then throw away these temporary strings. 

 But during pickling the memory-consumption was increasing and 
increasing.  I finally stopped it, when the memory-footprint had 
grown large enough to get the machine into constant swapping.
(the machine has 320MB ram in it, Python grew to about 344MB before
I killed it. At that time not even a tenth of the whole list had been
pickled.)

Is pickling more complicated internally, than I expected it to be ?


Another problem:
  when I create a few objects , (just enough to see memory consumption go
up for python in "top"), and then release them ("del"), then I see memory
consumption go down (almost) to the old size.
  But, when I create a really large list, and del it, then the memory-
footprint remains large. 

Is this a bug or a feature ?

(seen with 2.2.0, not yet tried with others)

-- 
Newsflash: Sproingy made it to the ground !
  read more ... <http://avl.enemy.org/sproingy/>