Reading in large logfiles, and processing lines in batches - maximising throughput?

[Chris Angelico]

On Wed, Sep 16, 2015 at 7:27 PM, Victor Hooi <victorhooi at gmail.com> wrote:
> Also, I originally used grouper because I thought it better to process lines in batches, rather than individually. However, is there actually any throughput advantage from doing it this way in Python? Or is there a better way of getting better throughput?
>

I very much doubt it'll improve throughput; what you're doing there is
reading individual lines, then batching them up into blocks of 1000,
and then stepping through the batches. In terms of disk read
performance, you're already covered, because the file object should be
buffered; if you're not doing much actual work in Python, that's
probably where your bottleneck is. But keep in mind the basic rules of
performance optimization:

1) Don't.
2) For experts only: Don't yet.
3) Measure first.

If you remember only the first rule, you're going to be correct most
of the time. Write your code to be idiomatic and clean, and *don't
worry* about performance. The second rule comes into play once you
have a fully working program, and you find that it's running too
slowly. (For example, you run "cat filename >/dev/null" and it takes
half a second, but you run your program on the same input file and it
takes half a day.) Okay, so you know your program needs some work. But
which parts of it are actually taking the time? If you just stare at
your code and make a guess, *you will be wrong*. So you follow the
third rule: Add a boatload of timing marks to the code. They'll slow
it down, of course, but you'll usually find that large slabs of the
code are so fast you can't even measure the time they're taking, so
there's no point optimizing them in any way. Only once you've proven
(a) that your program is "too slow" (for some measure of "slow"), and
(b) that it's _this part_ that's taking the bulk of the time, *then*
you can start improving performance.

So get rid of the grouper; it's violating all three rules. Give the
program a try without it, and see if you actually have a problem at
all. Maybe you don't!

ChrisA