From logging to files to a better solution: syslog, Sentry, Logstash, ....

[marco.nawijn at colosso.nl]

On Friday, September 11, 2015 at 9:22:42 AM UTC+2, Thomas Güttler wrote:
> Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
> > Thomas Güttler writes:
> > > ...
> > > Why we are unhappy with logging to files:
> > >
> > >  - filtering: We don't want to get INFO messages over the VPN.
> > 
> > You can quite easily control at what level messages are logged with
> > the standard Python logging framework. Each handler has a level
> > and will ignore messages at a lower level.
> 
> 
> I want INFO to be logged and stored on the remote host.
> Therefore I must not filter INFO messages.
> 
> I don't want to pull INFO messages over the VPN.
> 
> Ergo, the filtering at Python level does not help in my use case.
> Or I am missing something.
> 
> And now I have an ugly soup. 
> 
> The ugly soup is a text file with not well defined syntax. It looks line
> based. But sometimes there are log messages which span multiple lines ....
> 
> Remember: "Life is too short to (re)write parsers"
> 
> Yes, there are tools to parse that soup. But why create this soup in 
> the first place?
> 
> That's why I want to move from file based logging to a different solution.
> 
> Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(
> 
> 
> 
> > >  - Rotating: Rotating files is possible, but somehow cumbersome.
> > 
> > There are standard tools to rotate logfiles.
> > 
> > >  - Support structured logging of values (json) in the future.
> > 
> > Again, the Python logging framework is quite flexible with
> > respect to the format of logged messages.
> > 
> > > ...
> > > Which solution could fit for our environment?
> > 
> > I work for a customer with a similar environment (he uses "Zope" instead
> > of "Django") - and he uses logfiles. The logfiles are automatically
> > rotated and there are in the order of half a dozen to a dozen logfiles
> > per day.
> > 
> > When I have to analyse a problem with the help of the logfiles,
> > I do not copy them via VPN but do the filtering remotely and only
> > copy the filtered portion, if necessary.
> 
> Good to know that I am not the only one running servers in remote intranets.
> 
> Regards,
>   Thomas Güttler

So, if logging to json is on the horizon anyway, why don't you create
something like a MongoDb handler in the standard Python logging framework 
and run a MongoDb server in the client intranet? You could then connect
over VPN to MongoDb, filter the warning/error messages there on the server
side and fetch them to your local systems for analysis.

Marco