looping over a big file

[Jp Calderone]

On Sun, 3 Jul 2005 23:52:12 +0200, martian <nospam at hetnet.nl> wrote:
>Hi,
>
>I've a couple of questions regarding the processing of a big text file
>(16MB).
>
>1) how does python handle:
>
>> for line in big_file:
>
>is big_file all read into memory or one line is read at a time or a buffer
>is used or ...?

It uses an internal buffer to reach a happy medium between performance and memory usage.

>
>2) is it possible to advance lines within the loop? The following doesn't
>work:
>
>> for line in big_file:
>         line_after  = big_file.readline()
>

Yes, but you need to do it like this:

  fileIter = iter(big_file)
  for line in fileIter:
      line_after = fileIter.next()

  Don't mix iterating with any other file methods, since it will confuse the buffering scheme.

Jp