Multiprocessing & Logging

[Vinay Sajip]

Thibaut <merwin.irc <at> gmail.com> writes:

> Ok, I understand what happenned. In fact, configuring the logging before 
> forking works fine. Subprocess inherits the configuration, as I thought.
> 
> The problem was that I didn't passed any handler to the QueueListener 
> constructor. The when the listener recieved an message, it wasn't handled.
> 
> I'm not sure how the logging module works, but what handlers should I 
> pass the QueueListener constructor ? I mean, maybe that I would like 
> some messages (depending of the logger) to be logged to a file, while 
> some others message would just be printed to stdout.
> 
> This doesn't seem to be doable with a QueueListener. Maybe I should 
> implement my own system, and pass a little more informations with the 
> record sent in the queue : the logger name for example.
> 
> Then, in the main process I would do a logging.getLogger(loggername) and 
> log the record using this logger (by the way it was configured).
> 
> What do you think ?
> 

You probably need different logging configurations in different processes. In
your multiprocessing application, nominate one of the processes as a logging
listener. It should initialize a QueueListener subclass which you write. All
other processes should just configure a QueueHandler, which uses the same queue
as the QueueListener.

All the processes with QueueHandlers just send their records to the queue. The
process with the QueueListener picks these up and handles them by calling the
QueueListener's handle() method.

The default implementation of QueueListener.handle() is:

    def handle(self, record):
        record = self.prepare(record)
        for handler in self.handlers:
            handler.handle(record)

where self.handlers is just the handlers you passed to the QueueListener
constructor. However, if you want a very flexible configuration where different
loggers have different handlers, this is easy to arrange. Just configure logging
in the listener process however you want, and then, in your QueueListener
subclass, do something like this:

class MyQueueListener(logging.handlers.QueueListener):
    def handle(self, record):
        record = self.prepare(record)
        logger = logging.getLogger(record.name)
        logger.handle(record)

This will pass the events to whatever handlers are configured for a particular
logger.

I will try to update the Cookbook in the logging docs with this approach, and a
working script.

Background information is available here: [1][2]

Regards,

Vinay Sajip

[1]
http://plumberjack.blogspot.co.uk/2010/09/using-logging-with-multiprocessing.html
[2]
http://plumberjack.blogspot.co.uk/2010/09/improved-queuehandler-queuelistener.html