Python does not take up available physical memory

[Thomas Rachel]

Am 19.10.2012 21:03 schrieb Pradipto Banerjee:

> Thanks, I tried that.

What is "that"? It would be helpful to quote in a reasonable way. Look
how others do it.


> Still got MemoryError, but at least this time python tried to use the
> physical memory. What I noticed is that before it gave me the error
> it used up to 1.5GB (of the 2.23 GB originally showed as available) -
> so in general, python takes up more memory than the size of the file
> itself.

Of course - the file is not the only thing to be held by the process.

I see several approaches here:

* Process the file part by part - as the others already suggested, 
line-wise, but if you have e.g. a binary file format, other partings may 
be suitable as well - e.g. fixed block size, or parts given by the file 
format.

* If you absolutely have to keep the whole file data in memory, split it 
up in several strings. Why? Well, the free space in virtual memory is 
not necessarily contiguous. So even if you have 1.5G free, you might not 
be able to read 1.5G at once, but you might succeed in reading 3*0.5G.



Thomas