[Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface

[Guido van Rossum]

Ben: I wrote a long reply, it is inline below. However it's possible
that I am not seeing the use case right. Your proposal is written in
very general terms; perhaps you can come up with a more specific
example to defend it further? The UDP example somehow doesn't seem
very compelling to me.

On Sat, Feb 2, 2013 at 12:10 PM, Ben Darnell <ben at bendarnell.com> wrote:
> The event loop interface in PEP 3156 has an extensibility problem.  It seems
> likely that it will have a method like listen_udp() by the time it's done,
> but suppose that doesn't make it into the first official release.
> Third-party event loop implementations may want to provide UDP support as an
> extension, but the most consistent way to provide that extension is by
> adding new methods on the event loop object, where various extensions risk
> conflicting with each other or with new methods that become standardized
> later.

This may be based on a misunderstanding. If a specific event loop
implementation wants to offer a new transport, there is no reason to
add it as a method to the event loop. The app that wants to use that
transport has to import that event loop too, so the app might as well
call a module-level function that's specific to that event loop, in
order to instantiate the new transport.

We may have to point this out in the PEP, since it is likely that
implementers wanting to offer new features will think of adding new
methods to their event loop first. But that's a precious namespace
(since it's shared by all event loop implementations), whereas their
own implementation's module namespace is less precious.

> The PEP specifies the add_reader family of methods in part so that core
> protocol implementations can be shared across all EventLoop implementations
> that support these methods.  However, if those transports are implemented in
> a common base class (like Twisted's PosixReactorBase), there is no way for
> third-party transports to take advantage of a similar structure.

I'm not sure I understand this (the "there is no way" part). Is the
problem that that base class is private, or that the add_reader
methods may not be there, or what?

> I'd like
> to make it possible for transports to be developed independent of a
> particular EventLoop implementation in a way that is consistent with the way
> the core transports work.

Hm, now we're talking about something else. Transports implemented
*independently* from an event loop implementation should not assume
more than the standardized API. This feels rather limiting unless
we're talking about transports built on top of other protocols (e.g.
the mythical TCP-over-HTTP transport :-).

My expectation is that most transport implementations (as opposed to
protocols) are probably tied closely to an event loop implementation
(and note that the various Tulip event loop implementations using
select, poll, epoll, kqueue, are a single implementation for this
purpose -- OTOH the IocpEventLoop (in the iocp branch) is a different
implementation and, indeed, has different transport implementations!

> (This is a bigger concern for tulip than it is
> for twisted because twisted can update PosixReactorBase more frequently than
> the stdlib can change)

I think what you're really getting at here is that there is no 3rd
significant party cottage industry creating new transports. Transports
in practice are all part of the Twisted distribution, so development
is only gated by backwards compatibility requirements with user apps
(APIs, once offered, must remain supported and stable), not by
compatibilities with older versions of the rest of the framework (a
new transport introduced in Twisted 12.1 doesn't have to work with
Twisted 12.0).

> I propose turning the interface around so that transport creation uses
> static functions that take the event loop as an argument, with a
> double-dispatch mechanism to allow the event loop to provide the actual
> implementation when it can:
>
>   def create_connection(protocol_factory, host, port, event_loop=None):
>     if event_loop is None:
>       event_loop = get_event_loop()
>     # Note the use of a fully-qualified name in the registry
>     impl = event_loop.get_implementation('tulip.create_connection')
>     return impl(protocol_factory, host, port)

Hm. I don't see what this adds. It still always gets the protocol from
the event loop so moving this standardized method out of the event
loop class doesn't seem to buy anything. The implementation-dependent
work is just moved into get_implementation(). I also don't see why we
need a registry.

> New third-party transports could provide fallbacks for event loops that
> don't have their own implementations:
>
>   if impl is None:
>     # These supports_*() functions are placeholders for a to-be-determined
>     # introspection interface.
>     if supports_fd_interface(event_loop):
>       return posix_udp_implementation(*args)
>     elif supports_iocp_interface(event_loop):
>       return iocp_udp_implementation(*args)
>     else:
>       raise Exception("This transport is not supported on this event loop")

It seems you want each transport implementation to provide its own
create_connection() function, right? There's nothing wrong with that,
and I don't see that just because 3rd party transports will be
instantiated through a module-level function (in a specific module)
that means that the standard transports specified by the PEP (standard
in semantics, not in implementation!) can't be instantiated through
methods on the event loop.

Perhaps you are placing a higher value on consistency between standard
and non-standard transports? To me, it is actually positive to be
inconsistent here, so readers are made fully aware that a non-standard
transport is being used.

(The idea of introspection functions is fine, however.)

> Or they could plug into the event loop's implementation registry:
>
>   LibUVEventLoop.register_implementation('mymodule.listen_udp',
> libuv_udp_implementation)

Yeah, but wouldn't this likely be a private affair between UV's event
loop and UV's UDP transport, which are being distributed together?
Users who wish to use UDP (assuming PEP 3156 ends up not specifying
UDP support) are required to depend on a non-standard feature of their
event loop, you can't hide that with a registry. (Note that a UDP
transport has a different API than a TCP transport, and requires the
protocol to implement different methods as well.)

> This does introduce a little magic (is there any precedent for this kind of
> multiple-dispatch in the standard library?), but I like the way it keeps the
> event loop interface from getting too big and monolithic.  Third-party
> transports can avoid naming conflicts without looking fundamentally
> different from standard ones, and there's a clean path from doing something
> that's platform-specific (e.g. with add_reader and friends) to supporting
> multiple event loops to full standardization.

I actually consider it a good thing that when a concept is
standardized, the "name" of the API changes. When we adopt a 3rd party
module in the stdlib we typically give it a new name too, to avoid
confusion about which version is meant (since inevitably the 3rd party
has more variability than the version adopted into the stdlib).

With all that said, if a particular event loop implementation prefers
to add non-standard methods to their event loop object, and then
starts lobbying for its adoption in the standard, I can't stop them,
and they may even have a good shot at getting adopted in the next
Python version. But they should be aware of the risk they run, that
the next version of the stdlib might expose a different API under
their chose name. It will be easier for their users to transition if
they choose a way to spell their extension that is *not* likely to be
standardized, e.g. a function in their own module, or an event loop
method name with a custom prefix like uv_listen_udp().

I'm looking forward to explanations of why I am preventing the
developing of 3rd party transports with this response....

-- 
--Guido van Rossum (python.org/~guido)