optimizing memory utilization

[Istvan Albert]

> can't believe that python requires memory 200% "overhead" to store my
> data as lists of lists.  

And it almost certainly doesn't.

One one hand there could be a sizebale overhead when running
your program even when loading a single data line.
So don't extrapolate from a small datasets.

But what I believe happens to you is that you keep references
alive and you end up with data duplication.

For example if you do:

lines = file('whatever').readlines()

then you say for example:

stripped_lines = [ line.strip() for line in lines ]

you've just made your program use twice the memory
at that point of the code.

You may also inadvertently keep your 'lines' variable
in the scope of the full program  then
you'll effectively end up with a lot more memory consumption
overall.

Check and plug all these holes.

Istvan.