Multiprocessing & Logging

[Thibaut]

Le 07/04/2012 11:22, Thibaut DIRLIK a écrit :
> Hi,
>
> I'm currently writing a multiprocess applications with Python 3.2 and 
> multiprocessing module. My subprocesses will use a QueueHandler to log 
> messages (by sending them to the main process, which uses a 
> QueueListener). However, if logging is already configured when I 
> create the subprocesses, they will inherit the configuration, and 
> opened file descriptors used for logging in the main process.
>
> However, when I tried this with a basicConfig configuration, which 
> prints to a file, the messages are only written once to the file. I 
> don't understand why. Normally, the mainprocess contains a logging 
> handler to log to the file. This handler will be copied into the child 
> processes. Child processes will then have two handlers : a 
> QueueHandler, and a FileHandler. They should write to the file handler 
> and then send the message to the main process QueueListener, which 
> should write the message AGAIN to the FileHandler.
>
> But that's not the case. Any rea
>
> How can I totally "unset" the logging configuration in child processes 
> and only enable the QueueHandler (foor all configured loggers, not 
> only the root one) ?
>
> Thanks for your help,
>

Ok, I understand what happenned. In fact, configuring the logging before 
forking works fine. Subprocess inherits the configuration, as I thought.

The problem was that I didn't passed any handler to the QueueListener 
constructor. The when the listener recieved an message, it wasn't handled.

I'm not sure how the logging module works, but what handlers should I 
pass the QueueListener constructor ? I mean, maybe that I would like 
some messages (depending of the logger) to be logged to a file, while 
some others message would just be printed to stdout.

This doesn't seem to be doable with a QueueListener. Maybe I should 
implement my own system, and pass a little more informations with the 
record sent in the queue : the logger name for example.

Then, in the main process I would do a logging.getLogger(loggername) and 
log the record using this logger (by the way it was configured).

What do you think ?