threading and multiprocessing deadlock

[Martin Di Paola]

Hi!, in short your code should work.

I think that the join-joined problem is just an interpretation problem.

In pseudo code the background_thread function does:

def background_thread()
   # bla
   print("join?")
   # bla
   print("joined")

When running this function in parallel using threads, you will probably
get a few "join?" first before receiving any "joined?". That is because
the functions are running in parallel.

The order "join?" then "joined" is preserved within a thread but not
preserved globally.

Now, I see another issue in the output (and perhaps you was asking about 
this one):

join?
join?
myfnc
myfnc
join?
join?
joined.
joined.

So you have 4 "join?" that correspond to the 4 background_thread 
function calls in threads but only 2 "myfnc" and 2 "joined".

Could be possible that the output is truncated by accident?

I ran the same program and I got a reasonable output (4 "join?", "myfnc" 
and "joined"):

join?
join?
myfnc
join?
myfnc
join?
joined.
myfnc
joined.
joined.
myfnc
joined.

Another issue that I see is that you are not joining the threads that 
you spawned (background_thread functions).

I hope that this can guide you to fix or at least narrow the issue.

Thanks,
Martin.


On Mon, Dec 06, 2021 at 12:50:11AM +0100, Johannes Bauer wrote:
>Hi there,
>
>I'm a bit confused. In my scenario I a mixing threading with
>multiprocessing. Threading by itself would be nice, but for GIL reasons
>I need both, unfortunately. I've encountered a weird situation in which
>multiprocessing Process()es which are started in a new thread don't
>actually start and so they deadlock on join.
>
>I've created a minimal example that demonstrates the issue. I'm running
>on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
>([GCC 10.3.0] on linux).
>
>Here's the code:
>
>
>import time
>import multiprocessing
>import threading
>
>def myfnc():
>	print("myfnc")
>
>def run(result_queue, callback):
>	result = callback()
>	result_queue.put(result)
>
>def start(fnc):
>	def background_thread():
>		queue = multiprocessing.Queue()
>		proc = multiprocessing.Process(target = run, args = (queue, fnc))
>		proc.start()
>		print("join?")
>		proc.join()
>		print("joined.")
>		result = queue.get()
>	threading.Thread(target = background_thread).start()
>
>start(myfnc)
>start(myfnc)
>start(myfnc)
>start(myfnc)
>while True:
>	time.sleep(1)
>
>
>What you'll see is that "join?" and "joined." nondeterministically does
>*not* appear in pairs. For example:
>
>join?
>join?
>myfnc
>myfnc
>join?
>join?
>joined.
>joined.
>
>What's worse is that when this happens and I Ctrl-C out of Python, the
>started Thread is still running in the background:
>
>$ ps ax | grep minimal
> 370167 pts/0    S      0:00 python3 minimal.py
> 370175 pts/2    S+     0:00 grep minimal
>
>Can someone figure out what is going on there?
>
>Best,
>Johannes
>-- 
>https://mail.python.org/mailman/listinfo/python-list