[New-bugs-announce] [issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished

[Sergei Zobov]

New submission from Sergei Zobov <sergey at szobov.ru>:

In case if the process target function put something unpickable in the
`multiprocessing.Queue` and finish before `ForkingPickler`'s exception is handled we get two weird situations:
* Internal queue semaphore stay acquired and never will be released, so we get `qsize()` always returns 1(or more if more then one process) even if there is nothing in queue's buffer and `empty()` returns `True`.
* An exception's traceback will be silently suppressed.

To reproduce this behavior I wrote a test in cpython repo (be aware sometime it pass sometime no):
https://github.com/szobov/cpython/commit/95524f10a7b6510e9ab4647223cc67af85ebff86

I suppose it happens because of this:

* The function `util._exit_function()` calls every time process target function finished. Inside this function the global (for current process) flag `utils._exiting` marks as `True`:
https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py#L300
https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L294-L295
* When exception while pickling object happens in `Queue._feed` function we are ignore it because of the clause `is_exiting()` that checks the flag `utils._exiting` described above.
https://github.com/python/cpython/blob/master/Lib/multiprocessing/queues.py#L257-L259

So we have unreleased semaphore and ignored exceptions.
I think it's quite disappointing for user because they have broken queue and nothing about why it happened.

I'm not quite sure that it's a bag but I found nothing in documentation about it.

----------
messages: 329889
nosy: szobov
priority: normal
severity: normal
status: open
title: multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished
type: behavior
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35242>
_______________________________________