Daemonization / Popen / pipe issue

[John O'Hagan]

On Sat, 17 Mar 2012 18:17:10 -0400
Lee Clemens <java at leeclemens.net> wrote:

> On 03/16/2012 11:37 PM, John O'Hagan wrote:
> > On Fri, 16 Mar 2012 22:12:14 -0400
> > Lee Clemens <java at leeclemens.net> wrote:
> >
> >> I have a multi-threaded application
> >>
> >> I have provided a test-case here: https://gist.github.com/2054194
> > I haven't looked at your test case yet, but a possible cause is the fact
> > that the main application (or main thread) exits if it has finished
> > executing and only daemon threads remain. This can  abruptly terminate
> > threads which may be busy, for example, communicating via a pipe. The best
> > solution IMO is not to use daemon threads, but to give all threads a way to
> > terminate cleanly before the main thread does, even if this means using a
> > flag or the like.
> >
> > HTH,
> >
> > John
> >
> I call .join() on each spawned thread, whether it is daemonized or not.
> The code is available here:
> 
> https://gist.github.com/2054194

I've tried running your code (which at a glance looks very well-written) but I
get the broken pipe errors whether the DAEMONIZE flag is True or False
(Python 2.7, Debian testing). I don't have time to dig much deeper. 

> Is it generally not recommended to daemonize Python applications using
> this method? 

I'm no guru, but not that I know of. I haven't seen the os.fork method you use
before; out of interest, why not use Thread.setDaemon? 

> Do you have any references or data to support your opinion?
> Do you have any references which specifically state - "Daemonizing
> Python applications cause Python to work in unexpected ways."? 

Not particularly, although there are a few oldish references to "Python daemon
threads considered harmful" out there. You probably know that daemonising can
cover a multitude of sins, and it's always a good idea to ensure clean
termination and not just assume that what happens in a daemon thread on program
exit is moot. I'm not saying you haven't done that, in fact your errors seem to
occur during execution, not termination.

I replied to your post because I've had broken pipe errors before with
daemon threads communicating on sockets which sounded similar to your issue. I
solved them by rewriting so that the main thread told the threads to terminate,
and waited till they had. This made daemonising unnecessary in that case.


[...]

Regards,

John