Having trouble with tail -f standard input

[Derek Martin]

On Thu, Aug 21, 2008 at 02:58:24PM -0700, sab wrote:
> I have been working on a python script to parse a continuously growing
> log file on a UNIX server.

If you weren't aware, there are already a plethora of tools which do
this...  You might save yourself the trouble by just using one of
those.  Try searching for something like "parse log file" on google or
freshmeat.net or whatever...

> The input is the standard in, piped in from the log file.  The
> application works well for the most part, but the problem is when
> attempting to continuously pipe information into the application via
> the tail -f command.  The command line looks something like this:
> 
> tail -f <logfile> | grep <search string> | python parse.py

The pipe puts STDIN/STDOUT into "fully buffered" mode, which results
in the behavior you're seeing.  You can set the buffering mode of
those files in your program, but unfortunately tail and grep are not
your program...  You might get this to work by setting stdin to
non-blocking I/O in your Python program, but I don't think it will be
that easy...

You can get around this in a couple of ways.  One is to call tail and
grep from within your program, using something like os.popen()...
Then set the blocking mode on the resulting files.  You'll have to
feed the output of one to the input of the other, then read the output
of grep and parse that.  Yucky.  That method isn't very efficient,
since Python can do everything that tail and grep are doing for you...
So I'd suggest you read the file directly in your python program, and
use Python's regex parsing functionality to do what you're doing with
grep.  

As for how to actually do what tail does, I'd suggest looking at the
source code for tail to see how it does what it does.
 
But, if I were you, I'd just download something like swatch, and be
done with it. :)

-- 
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20080821/b0ffee12/attachment.sig>