Multiprocessing Queue strange behavior

[MRAB]

On 15/09/2010 21:10, Bruno Oliveira wrote:
> Hi list,
>
> I recently found a bug in my company's code because of a strange
> behavior using multiprocessing.Queue. The following code snippet:
>
> from multiprocessing import Queue
>
> queue = Queue()
> queue.put('x')
> print queue.get_nowait()
> Fails with:
>
> ...
>    File
> "E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py",
> line 153, in getNoWait
>      return self.get(False)
>    File
> "E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py",
> line 129, in get
>      raise Empty
> Queue.Empty
>
> Strangely, changing this to:
>
> queue = Queue()
> queue.put('x')
> time.sleep(0.1) # <<<
> print queue.get_nowait()
> Works as expected. Using the original snippet changing the import to
> threading's Queue also works.
>
> It seems like there's a bug in multiprocessing's Queue implementation.
> Opinions?
>
I don't think it's a bug as such.

The purpose of the multiprocessing queue is to transfer data between
different processes, which don't have a shared address space (unlike
threads, which do).

The transfer involves passing the data between the processes via a
pipe. This is done in a background thread and takes some time to
complete, so the data won't appear immediately. It looks like it
doesn't matter that the putter and the getter happen to be in the same
process, possibly because no-one expected that someone would use a
multiprocessing queue within the same process like that, so it doesn't
check for a shortcut.