[issue34781] infinite waiting in multiprocessing.Pool

[Tomáš Bouda]

Tomáš Bouda <tomas.bouda at gmail.com> added the comment:

It's very difficult to reproduce.

In this example to get stuck on 3.6/OSX I need to attach debugger. However, the freeze happens regardless of print/logging, even def f(): pass can get stuck. os.write() made no difference and frozen, as well, as I've just tried.

It might be even possible that there's another problem in debugger itself, however on 2.7/RHEL the actual (production) code is embedded in unittest with "discover" mode, run from shell, no debugger attached. 

I couldn't reproduce it today morning. Later on afternoon in another script it occured 2 times in a row. In the past months, I have seen the problem under various conditions. Originally, I used ProcessPoolExecutor before, where it happened rather often, so I rewrote the code to use Pool directly, the problem is rare, now, but still occurs.

The production code has many variants including logging/custom prints/no prints at all, from time to time it happens regardless of anything else. However, there's also a heavy load and high OS resource demand (tens of workers, tens of GBs read/allocated, many/explicit calls to GC, etc.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34781>
_______________________________________