Fast forward-backward (write-read)

[Steven D'Aprano]

On Wed, 24 Oct 2012 01:23:58 -0400, Dennis Lee Bieber wrote:

> On Tue, 23 Oct 2012 16:35:40 -0700, emile <emile at fenx.com> declaimed the
> following in gmane.comp.python.general:
> 
>> On 10/23/2012 04:19 PM, David Hutto wrote:
>> > forward =  [line.rstrip('\n') for line in f.readlines()]
>> 
>> f.readlines() will be big(!) and have overhead... and forward results
>> in something again as big.
>>
> 	Well, since file objects are iterable, could one just drop the
> .readlines() ? ( ... line in f )

Yes, but the bottleneck is still that the list comprehension will run to 
completion, trying to process the entire 100+ GB file in one go.

[...]
> 	And since the line-ends have already been stripped from forward,
> backward should just be:
> 
> 	backward = reversed(forward)

reversed returns a lazy iterator, but it requires that forward is a non-
lazy (eager) sequence. So again you're stuck trying to read the entire 
file into RAM.

-- 
Steven