High memory usage - program mistake or Python feature?

[Ben S]

Gerald Klix wrote:
> Simply call xreadlines again, that gives a new iterator.

How would I do it transparently, though? I mean, I want to have several
bits of code such as the following:

selectedLines = GetLinesThatMatch(allLines)

And I don't want to change them all if I need to switch between an
in-memory copy of the file (for speed) and the on-disk version from
xreadlines() (for memory conservation). Just using xreadlines again
means I have to change all these lines of code, as I can't just say
allLines = file.xreadlines() as it won't reset it each time.

I know this may sound like a trivial problem but I am trying to learn
how I can use Python to isolate myself from such changes.

--
Ben Sizer
http://pages.eidosnet.co.uk/kylotan


> Ben S wrote:
>> Hmm, in quick experiments with using xreadlines instead of readlines,
>> there is obviously the problem that while a single iteration over
>> either container works the same way, in order to repeat iterations
>> over xreadlines I need to somehow reset the iterator, which a quick
>> look at the documentation doesn't show me how to do. How do I do
>> this, so that my functions can take a list of lines without caring
>> whether those lines are in memory or coming from xreadlines?
>>
>> --
>> Ben Sizer
>> http://pages.eidosnet.co.uk/kylotan