Cropping log files

[Robin Munn]

Kamus of Kadizhar <yan at NsOeSiPnAeMr.com> wrote:
> Yet another possible newbie question.
>
> I'm tyring to figure out how to best crop a log file.  I have a file 
> that grows infinitely in length.  I want to keep only the last n entries 
> in the file.
> 
[snip]
>
> I don't want to suck the whole file into memory if I can help it, and I 
> can't help thinking that doing a
> for line in file(logfile)
>     nr += 1
>
> to count the number of lines, then reading the file again, discarding 
> the first nr - n records, writing the rest to a temp file, and then 
> renaming the files is the most efficient way to go.

Are you in a Unix environment here? If so, why not let the 'tail'
command do the work for you? Something like this:

    # Close any open file objects pointing to the log file
    long_log_filename = '/var/log/lotsoflines.log'
    short_log_filename = '/var/log/lotsoflines.log.truncated'
    lines_to_keep = 250
    cmd = 'tail -%(lines)d %(long)s > %(short)s && mv %(short)s %(long)s' % {
                  'lines': lines_to_keep,
                  'short': short_log_filename,
                  'long':  long_log_filename,
          }
    os.system(cmd)
    # Now reopen your file objects

Or if you really want to do it in pure Python, then have a look at the
source for 'tail' and see how it finds the last N lines. In most of my
experiments, I've found that 'tail' grabs the last 10 lines out of a
1-gigabyte log file within milliseconds, while 'wc -l' takes a lot
longer to count the whole file. That's not very scientific evidence, of
course, but why don't you try it and see for yourself?

-- 
Robin Munn
rmunn at pobox.com