[New-bugs-announce] [issue20147] multiprocessing.Queue.get() raises queue.Empty exception if even if an item is available

[Torsten Landschoff]

New submission from Torsten Landschoff:

The behaviour of multiprocessing.Queue surprised me today in that Queue.get() may raise an exception even if an item is immediately available. I tried to flush entries without blocking by using the timeout=0 keyword argument:
$ /opt/python3/bin/python3
Python 3.4.0b1 (default:247f12fecf2b, Jan  6 2014, 14:50:23) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python3/lib/python3.4/multiprocessing/queues.py", line 107, in get
    raise Empty
queue.Empty

Actually even passing a small non-zero timeout will not give me my queue entry:
>>> q.get(timeout=1e-6)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/torsten/opensrc/cpython/Lib/multiprocessing/queues.py", line 107, in get
    raise Empty
queue.Empty

Expected behaviour for me would be to return the item that is in the queue. I know that there is a kwarg *block* which gives me the desired behaviour:
>>> q.get(block=False)
'hi'
In my case the get call is embedded in my own module which does not currently expose the block parameter. My local solution is of course to update the wrapper:

if timeout == 0:
    timeout = None
    block = False

However I see a few smells here in the python standard library. First, everything else seems to accept timeout=0 as nonblocking:

>>> import threading
>>> lock = threading.Lock()
>>> lock.acquire(timeout=0)
True
>>> from queue import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
'hi'
Of special note is that queue.Queue behaves as I would have expected. IMHO it should be consistent with multiprocessing.Queue.

Also note that queue.Queue.get() and queue.Queue.put() name their blocking flag "block", while everybody else uses "blocking".

As a side note, I think the current approach is flawed in computing the deadline. Basically it does the following:

    deadline = time.time() + timeout
    if not self._rlock.acquire(block, timeout):
        raise Empty
    timeout = deadline - time.time()
    if timeout < 0 or not self._poll(timeout):
        raise Empty

On my system, just taking the time twice and computing the delta takes 2 microseconds:

>>> import time
>>> t0 = time.time(); time.time() - t0
2.384185791015625e-06

Therefore calling Queue.get(block, timeout) with 0 < timeout < 2e-6 will never return anything from the queue even though Queue.get(block=False) would do that. This contradicts the idea that Queue.get(block=False) will return faster than with block=True with any timeout > 0.

Apart from that, as Python does not currently support waiting on multiple sources, we currently often check a queue with a small timeout concurrently with doing other stuff. In case the system get really loaded, I would expect this to cause problems because the updated timeout may fall below zero.

Suggested patch attached.

----------
components: Library (Lib)
files: queue_timeout_0.diff
keywords: patch
messages: 207443
nosy: torsten
priority: normal
severity: normal
status: open
title: multiprocessing.Queue.get() raises queue.Empty exception if even if an item is available
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file33327/queue_timeout_0.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20147>
_______________________________________