Python does not take up available physical memory

[Chris Angelico]

On Sat, Oct 20, 2012 at 9:22 AM, Pradipto Banerjee
<pradipto.banerjee at adainvestments.com> wrote:
> Dennis,
>
> 1. Yes, .readlines() work where .read() fails. Thanks for the suggestion - this has really given a big boost to the size of the data I can read.

If at all possible, consider reading the file iteratively and
retaining only the current line. You say you need to "compare one line
versus another" - are you able, for instance, to sort the file
(externally; I think every OS these days will have a sort utility that
can handle files >RAM) and then compare the current line against a
small amount of retained state? This sort (pun intended) of trick can
not only get you past the limit, it will often improve running time
enormously too.

ChrisA