RotatingFileHandler bugs/errors and a general logging question.

[nicholas.petrella at gmail.com]

On May 9, 12:37 am, Vinay Sajip <vinay_sa... at yahoo.co.uk> wrote:
> On May 9, 12:52 am, nicholas.petre... at gmail.com wrote:> The infrastructure in which I am work needs the ability to have log
> > files written to from multiple instances of the same script and
> > potentially from hundreds or more different machines.
>
> > I know that the documentation suggests using a networkloggingserver
> > but I wanted to know if anyone had any other solutions to allow us to
> > build off of the current pythonloggingpackages.
>
> Dennis is right - the logging system is threadsafe but not safe
> against multiple processes (separate Python instances) writing to the
> samefile. It certainly sounds like you need a scalable solution - and
> having each script send the events to a network logging server seems a
> good way of handling the scalability requirement. The logger name used
> can include the script instance and machine name, e.g. by starting
> with hostname.scriptname.scriptpid... The socket server which receives
> the events can demultiplex them based on this information and write
> them to a central repository in any arrangement you care to implement
> (e.g. into onefileor several).
>
> Given that the example in the docs is a (basic) working example, is
> there any particular reason why you don't want to follow the suggested
> approach?
>
> Regards,
>
> Vinay Sajip


Our biggest concerns with the network solution is having a single
point of failure and the need for scalability. We could have
potentially thousands of machines doing logging across multiple
geographic sites. Our fear with the network solution is overwhelming
the server or group of servers as well as having a single point of
failure for the logging interface. In addition using a server or
servers would require added support for the logigng server machines.
Our NFS infrastructure is very well supported and can handle the load
generated by these machines already (A load which would be many times
more than what the logging would generate) which is why we would like
to log directly to file system without going through a separate
server. Also adding in a logging server introduces one more level
where we could potentially have failure. We would like to keep the
infrastructure for our logging as simple as possible as we rely on log
files to give us critical information when troubleshooting issues.

It sounds like my only option may be using a server in order to handle
the logging from different hosts. That or possibly having individual
log files for each host.

Thanks for your input. It is much appreciated.

-Nick