Reading log and saving data to DB

[marduk at python.net]

On Wed, Aug 14, 2013, at 09:18 AM, Guy Tamir wrote:
> Hi all,
> 
> I have a Ubuntu server running NGINX that logs data for me.
> I want to write a python script that reads my customized logs and after 
> a little rearrangement save the new data into my DB (postgresql).
> 
> The process should run about every 5 minutes and i'm expecting large
> chunks of data on several 5 minute windows..
> 
> My plan for achieving this is to install python on the server, write a
> script and add it to cron.
> 
> My question is what the simplest way to do this? 
> should i use any python frameworks? 

Rarely do I put "framework" and "simplest way" in the same set.

I would do 1 of 2 things:

* Write a simple script that reads lines from stdin, and writes to the
db.  Make sure it gets run in init before nginx does and tail -F -n 0 to
that script.  Don't worry about the 5-minute cron.

* Similar to above but if you want to use cron also store in the db the
offset of the last byte read in the file, then when the cron job kicks
off again seek to that position + 1 and begin reading, at EOF write the
offset again.

This is irrespective of any log rotating that is going on behind the
scenes, of course.