[Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface

[Ben Darnell]

On Sun, Feb 3, 2013 at 12:30 PM, Guido van Rossum <guido at python.org> wrote:

> Ben: I wrote a long reply, it is inline below. However it's possible
> that I am not seeing the use case right. Your proposal is written in
> very general terms; perhaps you can come up with a more specific
> example to defend it further? The UDP example somehow doesn't seem
> very compelling to me.
>

UDP is a real-life example from tornado  - we don't have any built-in
support for UDP, but people who need it have been able to build it without
touching tornado itself.  The same argument would apply to pipes or any
number of other (admittedly much more esoteric) network protocols.  I'll
elaborate on the UDP example below.


>
> On Sat, Feb 2, 2013 at 12:10 PM, Ben Darnell <ben at bendarnell.com> wrote:
> > The event loop interface in PEP 3156 has an extensibility problem.  It
> seems
> > likely that it will have a method like listen_udp() by the time it's
> done,
> > but suppose that doesn't make it into the first official release.
> > Third-party event loop implementations may want to provide UDP support
> as an
> > extension, but the most consistent way to provide that extension is by
> > adding new methods on the event loop object, where various extensions
> risk
> > conflicting with each other or with new methods that become standardized
> > later.
>
> This may be based on a misunderstanding. If a specific event loop
> implementation wants to offer a new transport, there is no reason to
> add it as a method to the event loop. The app that wants to use that
> transport has to import that event loop too, so the app might as well
> call a module-level function that's specific to that event loop, in
> order to instantiate the new transport.
>
> We may have to point this out in the PEP, since it is likely that
> implementers wanting to offer new features will think of adding new
> methods to their event loop first. But that's a precious namespace
> (since it's shared by all event loop implementations), whereas their
> own implementation's module namespace is less precious.
>

Right.  Third-party extensions to the event loop interface are inherently
problematic, so we'll have to provide them in some other way.  I'm
proposing a pattern for that "some other way" and then realizing that I
like it even for first-party interfaces.


>
> > The PEP specifies the add_reader family of methods in part so that core
> > protocol implementations can be shared across all EventLoop
> implementations
> > that support these methods.  However, if those transports are
> implemented in
> > a common base class (like Twisted's PosixReactorBase), there is no way
> for
> > third-party transports to take advantage of a similar structure.
>
> I'm not sure I understand this (the "there is no way" part). Is the
> problem that that base class is private, or that the add_reader
> methods may not be there, or what?
>

Suppose twisted did not have UDP support built in.  Most reactor
implementations subclass PosixReactorBase (with IOCPReactor as the notable
exception).  Twisted can add UDP support and implement listenUDP in
PosixReactorBase and IOCPReactor, and suddenly most reactors (even
third-party ones like TornadoReactor) support UDP for free.  Those that
don't (a hypothetical LibUVReactor?) can implement it themselves and
interoperate with everything else.

If a third party wanted to add UDP support separately from twisted's
release schedule, they can't do with an interface that is generically
usable across all reactors.  They could make a static function listenUDP()
that works with any IReactorFDSet, and maybe special-case IOCPReactor, but
then there'd be no way for a third-party LibUVReactor to participate.


>
> > I'd like
> > to make it possible for transports to be developed independent of a
> > particular EventLoop implementation in a way that is consistent with the
> way
> > the core transports work.
>
> Hm, now we're talking about something else. Transports implemented
> *independently* from an event loop implementation should not assume
> more than the standardized API. This feels rather limiting unless
> we're talking about transports built on top of other protocols (e.g.
> the mythical TCP-over-HTTP transport :-).



> My expectation is that most transport implementations (as opposed to
> protocols) are probably tied closely to an event loop implementation
> (and note that the various Tulip event loop implementations using
> select, poll, epoll, kqueue, are a single implementation for this
> purpose -- OTOH the IocpEventLoop (in the iocp branch) is a different
> implementation and, indeed, has different transport implementations!
>

add_reader is not very limiting except for its platform-specificity.  It's
possible to have a generic protocol across all posixy event loops and then
special-case the small number of interesting non-posixy ones (or maybe
there is some other class of methods that could be standardized for other
platforms?  Is there some set of methods analogous to add_reader that
multiple IOCP-based loops could share?)



>
> > (This is a bigger concern for tulip than it is
> > for twisted because twisted can update PosixReactorBase more frequently
> than
> > the stdlib can change)
>
> I think what you're really getting at here is that there is no 3rd
> significant party cottage industry creating new transports. Transports
> in practice are all part of the Twisted distribution, so development
> is only gated by backwards compatibility requirements with user apps
> (APIs, once offered, must remain supported and stable), not by
> compatibilities with older versions of the rest of the framework (a
> new transport introduced in Twisted 12.1 doesn't have to work with
> Twisted 12.0).
>

I was thinking more about release schedules.  Twisted has several releases
a year, but the standard library moves much more slowly.


>
> > I propose turning the interface around so that transport creation uses
> > static functions that take the event loop as an argument, with a
> > double-dispatch mechanism to allow the event loop to provide the actual
> > implementation when it can:
> >
> >   def create_connection(protocol_factory, host, port, event_loop=None):
> >     if event_loop is None:
> >       event_loop = get_event_loop()
> >     # Note the use of a fully-qualified name in the registry
> >     impl = event_loop.get_implementation('tulip.create_connection')
> >     return impl(protocol_factory, host, port)
>
> Hm. I don't see what this adds. It still always gets the protocol from
> the event loop so moving this standardized method out of the event
> loop class doesn't seem to buy anything. The implementation-dependent
> work is just moved into get_implementation(). I also don't see why we
> need a registry.
>

This version doesn't change much, it's mainly to set the stage for the
following variations.  However, it does have a few nice properties - it
keeps the (public) event loop interface small and manageable, and callers
don't need to touch actual event loop objects unless they want to have more
than one.  From a stylistic perspective I like this style of interface more
than using dozens of methods on the event loop object itself (even if those
dozens of methods are still there but hidden as an implementation detail).


>
> > New third-party transports could provide fallbacks for event loops that
> > don't have their own implementations:
> >
> >   if impl is None:
> >     # These supports_*() functions are placeholders for a
> to-be-determined
> >     # introspection interface.
> >     if supports_fd_interface(event_loop):
> >       return posix_udp_implementation(*args)
> >     elif supports_iocp_interface(event_loop):
> >       return iocp_udp_implementation(*args)
> >     else:
> >       raise Exception("This transport is not supported on this event
> loop")
>
> It seems you want each transport implementation to provide its own
> create_connection() function, right? There's nothing wrong with that,
> and I don't see that just because 3rd party transports will be
> instantiated through a module-level function (in a specific module)
> that means that the standard transports specified by the PEP (standard
> in semantics, not in implementation!) can't be instantiated through
> methods on the event loop.
>
> Perhaps you are placing a higher value on consistency between standard
> and non-standard transports? To me, it is actually positive to be
> inconsistent here, so readers are made fully aware that a non-standard
> transport is being used.
>

When third-party modules get absorbed into the standard library, it's often
possible to support both just by trying different imports until one works
(unittest.mock vs mock, json vs simplejson, etc).  Sometimes a module's
interface gets cleaned up and rearranged in the process, but that seems to
be less common.  It would be nice if a third-party transport could get
standardized and the only thing callers would need to change is their
imports.  However, this is a minor concern; as I wrote up this design I
realized I liked it for first-party work even if there were no third-party
modules to be consistent with.


>
> (The idea of introspection functions is fine, however.)
>
> > Or they could plug into the event loop's implementation registry:
> >
> >   LibUVEventLoop.register_implementation('mymodule.listen_udp',
> > libuv_udp_implementation)
>
> Yeah, but wouldn't this likely be a private affair between UV's event
> loop and UV's UDP transport, which are being distributed together?
>

But libuv doesn't necessarily contain the transport creation function.  The
idea is that someone can propose a transport interface in a third-party
module (mymodule.listen_udp in this example), implement it themselves for
some event loop implementations, and other event loops can declare
themselves compatible with it.

(And in an admittedly far-fetched scenario, if there were two third-party
UDP interfaces and LibUVEventLoop implemented one of them, yet another
party could build a bridge between the two, and then they'd plug it in with
register_implementation)



> Users who wish to use UDP (assuming PEP 3156 ends up not specifying
> UDP support) are required to depend on a non-standard feature of their
> event loop, you can't hide that with a registry. (Note that a UDP
> transport has a different API than a TCP transport, and requires the
> protocol to implement different methods as well.)
>

Yes, third-party transports will be non-standard, but in practice the
add_reader family makes it easy to get broad coverage in a quasi-standard
way, and the registry makes it possible to fill in the gaps.



>
> > This does introduce a little magic (is there any precedent for this kind
> of
> > multiple-dispatch in the standard library?), but I like the way it keeps
> the
> > event loop interface from getting too big and monolithic.  Third-party
> > transports can avoid naming conflicts without looking fundamentally
> > different from standard ones, and there's a clean path from doing
> something
> > that's platform-specific (e.g. with add_reader and friends) to supporting
> > multiple event loops to full standardization.
>
> I actually consider it a good thing that when a concept is
> standardized, the "name" of the API changes. When we adopt a 3rd party
> module in the stdlib we typically give it a new name too, to avoid
> confusion about which version is meant (since inevitably the 3rd party
> has more variability than the version adopted into the stdlib).
>

The *module* gets a new name (and that is indeed a good thing), but the
functions and classes within (usually) stay the same.


>
> With all that said, if a particular event loop implementation prefers
> to add non-standard methods to their event loop object, and then
> starts lobbying for its adoption in the standard, I can't stop them,
> and they may even have a good shot at getting adopted in the next
> Python version. But they should be aware of the risk they run, that
> the next version of the stdlib might expose a different API under
> their chose name. It will be easier for their users to transition if
> they choose a way to spell their extension that is *not* likely to be
> standardized, e.g. a function in their own module, or an event loop
> method name with a custom prefix like uv_listen_udp().
>

Agreed.


>
> I'm looking forward to explanations of why I am preventing the
> developing of 3rd party transports with this response....
>

I don't think the status quo prevents the development of third-party
transports, but it does implicitly encourage two bad habits:  A) adding
methods to the event loop, inviting name collisions, or B) just building on
add_reader and friends without thinking about non-posix platforms.  Of
course, no one expects a groundswell of third-party development at the
event loop and transport level, so this could just be so much
overengineering, but I like it from a stylistic perspective even without
the third-party benefits.

-Ben


>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/1212bcd4/attachment.html>