Python does not take up available physical memory

[Prasad, Ramit]

Emile van Sebille wrote:
> On 10/19/2012 10:08 AM, Pradipto Banerjee wrote:
> > Hi,
> >
> > I am trying to read a file into memory. The size of the file is around 1
> > GB. I have a 3GB memory PC and the Windows Task Manager shows  2.3 GB
> > available physical memory when I was trying to read the file. I tried to
> > read the file as follows:
> >
> >>>> fdata = open(filename, 'r').read()
> >
> > I got a "MemoryError". I was watching the Windows Task Manager while I
> > run the python command, and it appears that python **perhaps** never
> > even attempted to use more memory but gave me this error.
> >
> > Is there any reason why python can't read a 1GB file in memory even when
> > a 2.3 GB physical memory is available?
> 
> The real issue is likely that there is more than one copy of the file in
> memory somewhere.  I had a similar issue years back that I resolved by
> using numeric (now numpy?) as it had a more efficient method of
> importing content from disk.
> 
> Also realize that windows may not allow the full memory to user space.
> I'm not sure what exactly the restrictions are, but a 4Gb windows box
> doesn't always get you 4Gb of memory.
> 

Windows (by default) limits user space of a 32 bit machine to 2 GB.
This is a bit old but I think still applies to pre-Win7. 
(scroll down to "32-bit Client Effective Memory Limits" )
http://blogs.technet.com/b/markrussinovich/archive/2008/07/21/3092070.aspx

Offhand, I am not sure how this works in Win7, but for 32-bit clients
I doubt it has changed much.
This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.