Python does not take up available physical memory

[Prasad, Ramit]

Chris Angelico wrote:
> On Sat, Oct 20, 2012 at 4:08 AM, Pradipto Banerjee
> <pradipto.banerjee at adainvestments.com> wrote:
> > I am trying to read a file into memory. The size of the file is around 1 GB.
> > I have a 3GB memory PC and the Windows Task Manager shows  2.3 GB available
> > physical memory when I was trying to read the file. I tried to read the file
> > as follows:
> >
> >
> >
> >>>> fdata = open(filename, 'r').read()
> 
> Is this Python 2 or Python 3? Just throwing a random possibility out
> there, could it be that reading it in and converting it to Unicode
> text requires more memory than you have?
> 
> My recommendation: Unless you actually need to search the whole file
> as a single string, iterate over the file instead:
> 
> for line in open(filename):
>     # do something with line

If you (OP) are in Python 2.5+ I would do the following instead.

with open(filename) as f:
    for line in f:
        # do something with line

This will automatically close the file when it is done. I doubt
it will help with memory issues, but closing files after you
are done with them is a Good practice.


Ramit Prasad


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.