[issue19809] Python get stuck in second Popen call

[STINNER Victor]

STINNER Victor added the comment:

The creation of slave_popen is not protected by a lock, and so dummy_thread() may spawn a new process (master) at the same time than the main process (slave). If it occurs at the same time, the master may inherit a pipe of the slave process used internally by the subprocess module to send the exception to the parent process if a Python exception occurs in the child process.

The inheritance of pipes has been partially fixed in Python 3.2 with the new function _posixsubprocess.cloexec_pipe() which uses pipe2() function to atomically create a pipe with the O_CLOEXEC flag set.

The problem has been fixed complelty in Python 3.4 with the PEP 446: all files are now created non inheritable by default, and atomic functions to set the "non inheritable" flag are used when available.

You posted a Python 2.7 script, so I suppose that you are stuck at Python 2. In this case, you can workaround the issue by using a lock around the creation of any subprocess. To fix your example, just surround slave_popen creation with "with lock: ..." or lock.acquire()/lock.release(), as you did for master_popen.

I propose to convert this issue to a documentation issue: we should add a warning explaining that spawning processes in more than one thread can lead to such race condition and hang one or more threads. And suggest to use a lock to workaround such issue.

See also the atfork proposition which includes such lock:
http://bugs.python.org/issue16500

--

Your script has other issues:

- you should pass a file open in write mode for stdout/stderr
- you should close open(os.devnull) files, or open them outside the loop
- you should join the thread, or use a daemon thread. If you don't join threads, you will quickly read the limit of the number of threads
- you can use shell=True to avoid spawning two process to run sleep (or simply use time.sleep :-))
- you need a synchronization between the two threads to ensure that master_popen is created before trying to kill it

----------
nosy: +haypo, neologix, sbt

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19809>
_______________________________________