threading and multiprocessing deadlock

[Johannes Bauer]

Hi there,

I'm a bit confused. In my scenario I a mixing threading with
multiprocessing. Threading by itself would be nice, but for GIL reasons
I need both, unfortunately. I've encountered a weird situation in which
multiprocessing Process()es which are started in a new thread don't
actually start and so they deadlock on join.

I've created a minimal example that demonstrates the issue. I'm running
on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
([GCC 10.3.0] on linux).

Here's the code:


import time
import multiprocessing
import threading

def myfnc():
	print("myfnc")

def run(result_queue, callback):
	result = callback()
	result_queue.put(result)

def start(fnc):
	def background_thread():
		queue = multiprocessing.Queue()
		proc = multiprocessing.Process(target = run, args = (queue, fnc))
		proc.start()
		print("join?")
		proc.join()
		print("joined.")
		result = queue.get()
	threading.Thread(target = background_thread).start()

start(myfnc)
start(myfnc)
start(myfnc)
start(myfnc)
while True:
	time.sleep(1)


What you'll see is that "join?" and "joined." nondeterministically does
*not* appear in pairs. For example:

join?
join?
myfnc
myfnc
join?
join?
joined.
joined.

What's worse is that when this happens and I Ctrl-C out of Python, the
started Thread is still running in the background:

$ ps ax | grep minimal
 370167 pts/0    S      0:00 python3 minimal.py
 370175 pts/2    S+     0:00 grep minimal

Can someone figure out what is going on there?

Best,
Johannes