Python does not take up available physical memory

[Pradipto Banerjee]

Thanks, I tried that. Still got MemoryError, but at least this time python tried to use the physical memory. What I noticed is that before it gave me the error it used up to 1.5GB (of the 2.23 GB originally showed as available) - so in general, python takes up more memory than the size of the file itself.

-----Original Message-----
From: Python-list [mailto:python-list-bounces+pradipto.banerjee=adainvestments.com at python.org] On Behalf Of Emile van Sebille
Sent: Saturday, October 20, 2012 2:46 AM
To: python-list at python.org
Subject: Re: Python does not take up available physical memory

On 10/19/2012 10:08 AM, Pradipto Banerjee wrote:
> Hi,
>
> I am trying to read a file into memory. The size of the file is around 1
> GB. I have a 3GB memory PC and the Windows Task Manager shows  2.3 GB
> available physical memory when I was trying to read the file. I tried to
> read the file as follows:
>
>>>> fdata = open(filename, 'r').read()
>
> I got a "MemoryError". I was watching the Windows Task Manager while I
> run the python command, and it appears that python **perhaps** never
> even attempted to use more memory but gave me this error.
>
> Is there any reason why python can't read a 1GB file in memory even when
> a 2.3 GB physical memory is available?

The real issue is likely that there is more than one copy of the file in
memory somewhere.  I had a similar issue years back that I resolved by
using numeric (now numpy?) as it had a more efficient method of
importing content from disk.

Also realize that windows may not allow the full memory to user space.
I'm not sure what exactly the restrictions are, but a 4Gb windows box
doesn't always get you 4Gb of memory.

Emile


--
http://mail.python.org/mailman/listinfo/python-list

 This communication is for informational purposes only. It is not intended to be, nor should it be construed or used as, financial, legal, tax or investment advice or an offer to sell, or a solicitation of any offer to buy, an interest in any fund advised by Ada Investment Management LP, the Investment advisor.  Any offer or solicitation of an investment in any of the Funds may be made only by delivery of such Funds confidential offering materials to authorized prospective investors.  An investment in any of the Funds is not suitable for all investors.  No representation is made that the Funds will or are likely to achieve their objectives, or that any investor will or is likely to achieve results comparable to those shown, or will make any profit at all or will be able to avoid incurring substantial losses.  Performance results are net of applicable fees, are unaudited and reflect reinvestment of income and profits.  Past performance is no guarantee of future results. All financial data and other information are not warranted as to completeness or accuracy and are subject to change without notice.

Any comments or statements made herein do not necessarily reflect those of Ada Investment Management LP and its affiliates. This transmission may contain information that is confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.