Python does not take up available physical memory

[Emile van Sebille]

On 10/19/2012 10:08 AM, Pradipto Banerjee wrote:
> Hi,
>
> I am trying to read a file into memory. The size of the file is around 1
> GB. I have a 3GB memory PC and the Windows Task Manager shows  2.3 GB
> available physical memory when I was trying to read the file. I tried to
> read the file as follows:
>
>>>> fdata = open(filename, ‘r’).read()
>
> I got a “MemoryError”. I was watching the Windows Task Manager while I
> run the python command, and it appears that python **perhaps** never
> even attempted to use more memory but gave me this error.
>
> Is there any reason why python can’t read a 1GB file in memory even when
> a 2.3 GB physical memory is available?

The real issue is likely that there is more than one copy of the file in 
memory somewhere.  I had a similar issue years back that I resolved by 
using numeric (now numpy?) as it had a more efficient method of 
importing content from disk.

Also realize that windows may not allow the full memory to user space. 
I'm not sure what exactly the restrictions are, but a 4Gb windows box 
doesn't always get you 4Gb of memory.

Emile