Multiprocessing & Logging

[Thibaut]

Le 07/04/2012 16:47, Vinay Sajip a écrit :
> Thibaut<merwin.irc<at>  gmail.com>  writes:
>
>> Ok, I understand what happenned. In fact, configuring the logging before
>> forking works fine. Subprocess inherits the configuration, as I thought.
>>
>> The problem was that I didn't passed any handler to the QueueListener
>> constructor. The when the listener recieved an message, it wasn't handled.
>>
>> I'm not sure how the logging module works, but what handlers should I
>> pass the QueueListener constructor ? I mean, maybe that I would like
>> some messages (depending of the logger) to be logged to a file, while
>> some others message would just be printed to stdout.
>>
>> This doesn't seem to be doable with a QueueListener. Maybe I should
>> implement my own system, and pass a little more informations with the
>> record sent in the queue : the logger name for example.
>>
>> Then, in the main process I would do a logging.getLogger(loggername) and
>> log the record using this logger (by the way it was configured).
>>
>> What do you think ?
>>
> You probably need different logging configurations in different processes. In
> your multiprocessing application, nominate one of the processes as a logging
> listener. It should initialize a QueueListener subclass which you write. All
> other processes should just configure a QueueHandler, which uses the same queue
> as the QueueListener.
>
> All the processes with QueueHandlers just send their records to the queue. The
> process with the QueueListener picks these up and handles them by calling the
> QueueListener's handle() method.
>
> The default implementation of QueueListener.handle() is:
>
>      def handle(self, record):
>          record = self.prepare(record)
>          for handler in self.handlers:
>              handler.handle(record)
>
> where self.handlers is just the handlers you passed to the QueueListener
> constructor. However, if you want a very flexible configuration where different
> loggers have different handlers, this is easy to arrange. Just configure logging
> in the listener process however you want, and then, in your QueueListener
> subclass, do something like this:
>
> class MyQueueListener(logging.handlers.QueueListener):
>      def handle(self, record):
>          record = self.prepare(record)
>          logger = logging.getLogger(record.name)
>          logger.handle(record)
>
> This will pass the events to whatever handlers are configured for a particular
> logger.
>
> I will try to update the Cookbook in the logging docs with this approach, and a
> working script.
>
> Background information is available here: [1][2]
>
> Regards,
>
> Vinay Sajip
>
> [1]
> http://plumberjack.blogspot.co.uk/2010/09/using-logging-with-multiprocessing.html
> [2]
> http://plumberjack.blogspot.co.uk/2010/09/improved-queuehandler-queuelistener.html
>

This is exactly what I wanted, it seems perfect. However I still have a 
question, from what I understood,
I have to configure logging AFTER creating the process, to avoid 
children process to inherits the logging config.

Unless there is a way to "clean" logging configuration in children 
processes, so they only have one handler : the QueueHandler.

I looked at the logging code and it doesn't seems to have an easy way to 
do this. The problem of configuring the logging
after the process creation is that... I can't log during process 
creation. But if it's too complicated, I will just do this.

Thanks again for your help Vinay,