first, second, etc line of text file

[Jeff McNeil]

Yup, I actually had log files in mind, too.  I process Apache logs from an
8-way cluster every 15 minutes. There are 32,000 sites on said cluster;
that's a lot of log data!

I didn't actually think anyone would go *try* to
open("war_and_peace.txt").readlines().. I just meant it as a generalization
for "really large file." =)

-Jeff

On 7/25/07, Jay Loden <python at jayloden.com> wrote:
>
> Grant Edwards wrote:
> > On 2007-07-25, Jeff <jeffober at gmail.com> wrote:
> >
> >> That might be a memory problem if you are running multiple processes
> >> regularly, such as on a webserver.
> >
> > I suppose if you did it in parallel 50 processes, you could use
> > up 250MB of RAM.  Still not a big deal on many servers.  A
> > decent OS will swap regions that aren't being used to disk, so
> > it's likely not to be a problem.
>
> Or, you might be reading from a text file dramatically larger than a 3MB
> copy of War and Peace. I regularly deal with log files that are often many
> times that, including some that have been well over a GB or more. Trust me,
> you don't want to read in the entire file when it's a 1.5GB text file.
> It's true that many times readlines() will work fine, but there are also
> certainly cases where it's not acceptable for memory and performance
> reasons.
>
> -Jay
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20070725/03f1417e/attachment.html>