[New-bugs-announce] [issue33076] Trying to cleanly terminate a threaded Queue at exit of program raises an "EOFError"

[Adrien]

New submission from Adrien <delgan.py at gmail.com>:

Hi.

I use a worker Thread to which I communicate trough a multiprocessing Queue. I would like to properly close this daemon thread when my program terminates, so I registered a "stop()" function using "atexit.register()".

However, this raises an "EOFError" because the multiprocessing module uses "atexit.register()" too and closes the Queue internal pipe connections before that my thread ends.

After scratching inside the multiprocessing module, I tried to summarize my understanding of the problem here: https://stackoverflow.com/a/49244528/2291710

I joined a demonstration script that triggers the bug with (at least) Python 3.5/3.6 on both Windows and Linux.

The issue is fixable by forcing multiprocessing "atexit.register()" before mine with "import multiprocessing.queues", but this means I would rely on an implementation detail, and others dynamic calls made to "atexit.register()" (like one I saw in multiprocessing "get_logger()" for example) could break it again.
I first thought that "atexit.register()" could accept an optional "priority" argument, but every developers would probably want to be first. Could a subtle change be made however to guarantee that registered functions are executed before Python internal ones? As for now, the atexit statement "The assumption is that lower level modules will normally be imported before higher level modules and thus must be cleaned up later" is not quite true.

I do not know what to do with it, from what I know there is no way to achieve an automatic yet clean closure of such worker, so I would like to know if some kind of fix is possible for a future version of Python.

Thanks for your time.

----------
components: Library (Lib)
files: bug.py
messages: 313841
nosy: Delgan
priority: normal
severity: normal
status: open
title: Trying to cleanly terminate a threaded Queue at exit of program raises an "EOFError"
type: behavior
versions: Python 3.5, Python 3.6
Added file: https://bugs.python.org/file47486/bug.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33076>
_______________________________________