[Python-ideas] Tulip / PEP 3156 - subprocess events

[Paul Moore]

On 18 January 2013 08:38, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Transports are about the communications channel
> - network sockets
> - OS pipes
> - shared memory
> - CANbus
> - protocol tunneling
>
> Transports should only be platform specific at the base layer where
> they actually need to interact with the OS through the event loop.
> Higher level transports should be connected to lower level protocols
> based on APIs provided by those transports and protocols themselves.
>
> The *whole point* of the protocol vs transport model is to allow you
> to write adaptive stacks.

Interesting. On that basis, the whole subprocess interaction scenario
is not a low level transport at all (contrary to what I understood
from Guido's suggestion of an event loop method) and so should be
built in user code (OK, probably as a standard library helper, but
definitely not as specialist methods on the event loop) layered on the
low-level pipe transport.

That was my original instinct, but it fell afoul of

1. The Windows implementation of a low level pipe transport doesn't
exist (yet) and I don't know enough about IOCP to write it [1].
2. I don't understand the programming model well enough to understand
how to write a transport/protocol layer (coroutine head explosion
issue).

I have now (finally!) got Guido's point that implementing a process
protocol will give me a good insight into how this stuff is meant to
work. I'm still struggling to understand why he thinks it needs a
dedicated method on the event loop, rather than being a higher-level
layer like you're suggesting, but I'm at least starting to understand
what questions to ask.

Paul

[1] There is some stuff in the IOCP documentation about handles having
to be opened in OVERLAPPED mode, which worries me here as it may imply
that arbitrary pipes (such as the ones subprocess.Popen uses) can't be
plugged in. It's a bit like setting a filehandle to nonblocking in
Unix, but it has to be done at open time, IIUC. I think I saw an email
about this that I need to hunt out.