From logging to files to a better solution: syslog, Sentry, Logstash, ....

[Thomas Güttler]

Am Freitag, 11. September 2015 10:18:11 UTC+2 schrieb marco.... at colosso.nl:
> On Friday, September 11, 2015 at 9:22:42 AM UTC+2, Thomas Güttler wrote:
> > Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
> > > Thomas Güttler writes:
> > > > ...
> > > > Why we are unhappy with logging to files:
> > > >
> > > >  - filtering: We don't want to get INFO messages over the VPN.
> > > 
> > > You can quite easily control at what level messages are logged with
> > > the standard Python logging framework. Each handler has a level
> > > and will ignore messages at a lower level.
> > 
> > 
> > I want INFO to be logged and stored on the remote host.
> > Therefore I must not filter INFO messages.
> > 
> > I don't want to pull INFO messages over the VPN.
> > 
> > Ergo, the filtering at Python level does not help in my use case.
> > Or I am missing something.
> > 
> > And now I have an ugly soup. 
> > 
> > The ugly soup is a text file with not well defined syntax. It looks line
> > based. But sometimes there are log messages which span multiple lines ....
> > 
> > Remember: "Life is too short to (re)write parsers"
> > 
> > Yes, there are tools to parse that soup. But why create this soup in 
> > the first place?
> > 
> > That's why I want to move from file based logging to a different solution.
> > 
> > Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(
> > 
> > 
> > 
> > > >  - Rotating: Rotating files is possible, but somehow cumbersome.
> > > 
> > > There are standard tools to rotate logfiles.
> > > 
> > > >  - Support structured logging of values (json) in the future.
> > > 
> > > Again, the Python logging framework is quite flexible with
> > > respect to the format of logged messages.
> > > 
> > > > ...
> > > > Which solution could fit for our environment?
> > > 
> > > I work for a customer with a similar environment (he uses "Zope" instead
> > > of "Django") - and he uses logfiles. The logfiles are automatically
> > > rotated and there are in the order of half a dozen to a dozen logfiles
> > > per day.
> > > 
> > > When I have to analyse a problem with the help of the logfiles,
> > > I do not copy them via VPN but do the filtering remotely and only
> > > copy the filtered portion, if necessary.
> > 
> > Good to know that I am not the only one running servers in remote intranets.
> > 
> > Regards,
> >   Thomas Güttler
> 
> So, if logging to json is on the horizon anyway, why don't you create
> something like a MongoDb handler in the standard Python logging framework 
> and run a MongoDb server in the client intranet? You could then connect
> over VPN to MongoDb, filter the warning/error messages there on the server
> side and fetch them to your local systems for analysis.


Yes I could set up a MongoDB server. But I would like to reuse and not to reinvent. 

I would like to go a proven solid road.