execfile eats memory

[Fredrik Lundh]

Enrique Palomo Jiménez wrote:

> The information will be used no more than 3-4 days a
> month and install databases is not allowed.
> [...]
> Someone suggest me the pickle module. I'll try it and it
> works, but pickle dumps is slower than writing the file
> with pythonic syntax. (26 seconds vs 6)

did you try cPickle instead of pickle?

(also make sure that you're using binary pickles).

if you only use simple data types, the marshal module can sometimes
be faster than cPickle (it's always faster than pickle)

> After that, i have a DIC.txt to execfile with it. The application dies there.
> The memory grow up till de limit.
> Is possible that execfile of a 19 mb file takes more than 600 mb of memory?

15-20 bytes is more than enough; e.g.

     "*"*1000*1000*600

as for exec'ing huge files, it's not that unlikely that the Python compiler
may need 30 bytes per source byte to hold parser trees and byte code
structures.  to get exact figures, check the peak memory consumption
when you parse smaller (i.e. megabyte-sized) parts of your file.

</F>