[issue12328] multiprocessing's overlapped PipeConnection on Windows

[sbt]

New submission from sbt <shibturn at gmail.com>:

There are some problems with the new Windows overlapped implementation
of PipeConnection in the default branch.

1) poll(0) can return False when an empty string is in the pipe: if
   the next message in the pipe is b"" then PeekNamedPipe() returns
   (0, 0) which is indistiguishable from the case where the pipe is
   empty.

   This affects versions 2.6-3.2 of Python on Windows as well.  For
   old versions I would just document the fact that poll() will fail
   to see that there is data in the pipe if the next message is an
   empty string.  In practice this is not a big deal since pipes are
   primarily used for pickled objects which will never be the empty
   string.

2) _poll() forgets to check self._buffered, so it can return False
   even if there is buffered data.

3) Even if (2) is fixed, _poll() with a non-zero timeout can mess up
   message boundaries if the pipe contains messages of length zero or
   one.  To fix this overlapped_GetOverlappedResult() needs to be
   changed to not swallow and forget ERROR_MORE_DATA errors.

4) In _poll() there is a race condition: if the read operation
   completes after WaitForMultipleObjects() timesout, but before the
   operation is cancelled, then the data read will be discarded.

5) The code assumes that if CancelIo()/CancelIoEx() returns
   successfully then the OVERLAPPED structure and associated buffer
   may be deallocated immediately.  But the documentation says that if
   CancelIo()/CancelIoEx() succeeds then cancellation has only been
   *requested*:

     http://msdn.microsoft.com/en-us/library/aa363792%28v=vs.85%29.aspx

     If [CancelIoEx()] succeeds, the return value is nonzero. The
     cancel operation for all pending I/O operations issued by the
     calling thread for the specified file handle was successfully
     *requested*. The application must not free or reuse the
     OVERLAPPED structure associated with the canceled I/O operations
     until they have *completed*. The thread can use the
     GetOverlappedResult function to determine when the I/O operations
     themselves have been completed.

   We should wait for cancellation to *complete* by using
   GetOverlappedResult(..., TRUE), otherwise we risk a crash.  If the
   operation was cancelled then FALSE is returned with GetLastError()
   == ERROR_OPERATION_ABORTED.

   I would suggest making it a programming error for the overlapped
   object to be deallocated while the operation is still pending, and
   to print a RuntimeError if that happens.  (This does not prevent us
   from still trying to prevent a crash using CancelIoEx().)

6) Not a bug exactly, but poll(timeout) is no longer interruptible
   with Ctrl-C.  This also means that Queue.get() is no longer
   interruptible.

--

The unit tests attached pass on Unix.  On Windows versions up to 3.2,
only test_empty_string fails.  On Windows version 3.3 all three fail.

----------
components: Extension Modules
files: test_pipe_poll.py
messages: 138268
nosy: jnoller, pitrou, sbt
priority: normal
severity: normal
status: open
title: multiprocessing's overlapped PipeConnection on Windows
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file22345/test_pipe_poll.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12328>
_______________________________________