How to properly implement worker processes

[Dennis Jacobfeuerborn]

On Wednesday, August 22, 2012 7:46:34 PM UTC+2, Ian wrote:
> On Wed, Aug 22, 2012 at 11:29 AM, Dennis Jacobfeuerborn
> 
> <djacobfeuerborn at gmail.com> wrote:
> 
> > Hi,
> 
> > I'm trying to implement a system for periodically checking URLs and I've run into problems with some of the implementation details. The URLs are supposed to be checked continuously until the config for an URL is explicitly removed.
> 
> >
> 
> > The plan is to spawn a worker process for each URL that sends the status of the last check to its parent which keeps track of the state of all URLs. When a URL is no longer supposed to be checked the parent process should shutdown/kill the respective worker process.
> 
> >
> 
> > What I've been going for so far is that the parent process creates a global queue that is passed to all children upon creation which they use to send status messages to the parent. Then for each process a dedicated queue is created that the parent uses to issue commands to the child.
> 
> >
> 
> > The issue is that since the child processes spent some time in sleep() when a command from the parent comes they cannot respond immediately which is rather undesirable. What I would rather like to do is have the parent simply kill the child instead which is instantaneous and more reliable.
> 
> >
> 
> > My problem is that according to the multiprocessing docs if I kill the child while it uses the queue to send a status to the parent then the queue becomes corrupted and since that queue is shared that means the whole thing pretty much stops working.
> 
> >
> 
> > How can I get around this problem and receive status updates from all children efficiently without a shared queue and with the ability to simply kill the child process when it's no longer needed?
> 
> 
> 
> The usual approach to killing worker processes safely is to send them
> 
> an "exit" command, which they should respond to by terminating
> 
> cleanly.  Instead of using sleep(), have the workers do a blocking
> 
> get() on the queue with a timeout.  This way they'll receive the
> 
> "exit" message immediately as desired, but they'll still wake up at
> 
> the desired intervals in order to do their work.

I was thinking about something like that but the issue is that this really only works when you don't do any actual blocking work. I may be able to get around the sleep() but then I have to fetch the URL or do some other work that might block for a while so the get() trick doesn't work.
Also the child process might not be able to deal with such an exit command at all for one reason or another so the only safe way to get rid of it is for the parent to kill it.

The better option would be to not use a shared queue for communication and instead use only dedicated pipes/queues for each child process but the doesn't seem to be a way to wait for a message from multiple queues/pipes. If that were the case then I could simply kill the child and get rid of the respective pipes/queues without affecting the other processes or communication channels.

Regards,
  Dennis