[Python-ideas] async: feedback on EventLoop API

[Nick Coghlan]

On Tue, Dec 18, 2012 at 2:01 PM, Guido van Rossum <guido at python.org> wrote:

> On Mon, Dec 17, 2012 at 7:20 PM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:Also, event loop implementations are allowed to offer additional APIs
> on their implementation. If the need for multiple handlers per FD only
> exists on those platforms where the platform's event loop supports it,
> no harm is done if the functionality is only available through a
> platform-specific API.
>

Sure, but since we know this capability is offered by multiple event loops,
it would be good if there was a defined way to go about exposing it.


> But still, I don't understand the use case. Possibly it is using file
> descriptors as a more general signaling mechanism? That sounds pretty
> platform specific anyway (on Windows, FDs must represent sockets).
>
> If someone shows me a real-world use case I may change my mind.
>

The most likely use case that comes to mind is monitoring and debugging
(i.e. the event loop equivalent of a sys.settrace). Being able to tap into
a datastream (e.g. to dump it to a console or pipe it to a monitoring
process) can be really powerful, and being able to do it at the Python
level means you have this kind of capability even without root access to
the machine to run Wireshark.

There are other more obscure signal analysis use cases that occur to me,
but those could readily be handled with a custom transport implementation
that duplicated that data stream, so I don't think there's any reason to
worry about those.

 > Related, the protocol/transport API design may end up needing to consider
> > the gather/scatter problem (i.e. fanning out data from a single
> transport to
> > multiple consumers, as well as feeding data from multiple producers into
> a
> > single underlying transport). Actual *implementations* of such tools
> > shouldn't be needed in the standard suite, but at least understanding how
> > you would go about writing multiplexers and demultiplexers can be a good
> > test of a stacked I/O design.
>
> Twisted supports this for writing through its writeSequence(), which
> appears in Tulip and PEP 3156 as writelines(). (Though IIRC Glyph told
> me that Twisted rarely uses the platform's scatter/gather primitives,
> because they are so damn hard to use, and the kernel implementation
> often just joins the buffers together before passing it to the regular
> send()...)
>
> But regardless, I don't think scatter/gather would use multiple
> callbacks per FD.
>
> I think it would be really hard to benefit from reading into multiple
> buffers in Python.
>

Sorry, I wasn't quite clear on what I meant by gather/scatter and it's more
a protocol thing than an event loop thing.

Specifically, gather/scatter interfaces are most useful for multiplexed
transports. The ones I'm particularly familiar with are traditional
telephony transports like E1 links, with 15 time-division-multiplexed
channels on the wire (and a signalling timeslot), as well a few different
HF comms protocols. When reading from one of those, you have a
demultiplexing component which is reading the serial data coming in on the
wire and making it look like 15 distinct data channels from the
application's point of view. Similarly, the output multiplexer takes 15
streams of data from the application and interleaves them into the single
stream on the wire.

The rise of packet switching means that sharing connections like that is
increasingly less common, though, so gather/scatter devices are
correspondingly less useful in a networking context. The only modern use
cases I can think of that someone might want to handle with Python are
things like sharing a single USB or classic serial connection amongst
multiple data streams. However, I suspect the standard transport and
protocol API definitions already proposed should also suffice for the
gather/scatter use case, as such a component would largely work like any
other protocol-as-transport adapter, with the difference being that there
would be a many-to-one relationship between the number of interfaces on the
application side and those on the communications side.

(Technically, gather/scatter components can also be used the other way
around to distribute a single data stream across multi transports, but that
use case is even less likely to come up when programming in Python.
Multi-channel HF data comms is the only possibility that really comes to
mind)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121218/d73c5cbb/attachment.html>