From logging to files to a better solution: syslog, Sentry, Logstash, ....

[Thomas Güttler]

Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
> Thomas Güttler writes:
> > ...
> > Why we are unhappy with logging to files:
> >
> >  - filtering: We don't want to get INFO messages over the VPN.
> 
> You can quite easily control at what level messages are logged with
> the standard Python logging framework. Each handler has a level
> and will ignore messages at a lower level.


I want INFO to be logged and stored on the remote host.
Therefore I must not filter INFO messages.

I don't want to pull INFO messages over the VPN.

Ergo, the filtering at Python level does not help in my use case.
Or I am missing something.

And now I have an ugly soup. 

The ugly soup is a text file with not well defined syntax. It looks line
based. But sometimes there are log messages which span multiple lines ....

Remember: "Life is too short to (re)write parsers"

Yes, there are tools to parse that soup. But why create this soup in 
the first place?

That's why I want to move from file based logging to a different solution.

Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(



> >  - Rotating: Rotating files is possible, but somehow cumbersome.
> 
> There are standard tools to rotate logfiles.
> 
> >  - Support structured logging of values (json) in the future.
> 
> Again, the Python logging framework is quite flexible with
> respect to the format of logged messages.
> 
> > ...
> > Which solution could fit for our environment?
> 
> I work for a customer with a similar environment (he uses "Zope" instead
> of "Django") - and he uses logfiles. The logfiles are automatically
> rotated and there are in the order of half a dozen to a dozen logfiles
> per day.
> 
> When I have to analyse a problem with the help of the logfiles,
> I do not copy them via VPN but do the filtering remotely and only
> copy the filtered portion, if necessary.

Good to know that I am not the only one running servers in remote intranets.

Regards,
  Thomas Güttler