[Python-ideas] asyncore: included batteries don't fit

[Guido van Rossum]

On Tue, Oct 9, 2012 at 11:00 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
> Oh my me. This is a very long thread that I probably should have replied to
> a long time ago. This thread is intensely long right now, and tonight is the
> first chance I've had to try and go through it comprehensively. I'll try to
> reply to individual points made in the thread -- if I missed yours, please
> don't be offended, I promise it's my fault :)

No problem, I'm running behind myself...

> FYI, I'm the sucker who originally got tricked into starting PEP 3153, aka
> async-pep.

I suppose that's your pet name for it. :-) For most everyone else it's PEP 3153.

> First of all, I'm glad to see that there's some more "let's get that pep
> along" movement. I tabled it because:
>
> a) I didn't have enough time to contribute,
> b) a lot of promised contributions ended up not happening when it came down
> to it, which was incredibly demotivating. The combination of this thread,
> plus the fact that I was strong armed at Pycon ZA by a bunch of community
> members that shall not be named (Alex, Armin, Maciej, Larry ;-)) into
> exploring this thing again.
>
> First of all, I don't feel async-pep is an attempt at twisted light in the
> stdlib. Other than separation of transport and protocol, there's not really
> much there that even smells of twisted (especially since right now I'd
> probably throw consumers/producers out) -- and that separation is simply
> good practice. Twisted does the same thing, but it didn't invent it.
> Furthermore, the advantages seem clear: reusability and testability are more
> than enough for me.
>
> If there's one take away idea from async-pep, it's reusable protocols.

Is there a newer version that what's on
http://www.python.org/dev/peps/pep-3153/ ? It seems to be missing any
specific proposals, after spending a lot of time giving a rationale
and defining some terms. The version on
https://github.com/lvh/async-pep doesn't seem to be any more complete.

> The PEP should probably be a number of PEPs. At first sight, it seems that
> this number is at least four:
>
> 1. Protocol and transport abstractions, making no mention of asynchronous IO
> (this is what I want 3153 to be, because it's small, manageable, and
> virtually everyone appears to agree it's a fantastic idea)

But the devil is in the details. *What* specifically are you
proposing? How would you write a protocol handler/parser without any
reference to I/O? Most protocols are two-way streets -- you read some
stuff, and you write some stuff, then you read some more. (HTTP may be
the exception here, if you don't keep the connection open.)

> 2. A base reactor interface

I agree that this should be a separate PEP. But I do think that in
practice there will be dependencies between the different PEPs you are
proposing.

> 3. A way of structuring callbacks: probably deferreds with a built-in
> inlineCallbacks for people who want to write synchronous-looking code with
> explicit yields for asynchronous procedures

Your previous two ideas sound like you're not tied to backward
compatibility with Tornado and/or Twisted (not even via an adaptation
layer). Given that we're talking Python 3.4 here that's fine with me
(though I think we should be careful to offer a path forward for those
packages and their users, even if it means making changes to the
libraries). But Twisted Deferred is pretty arcane, and I would much
rather not use it as the basis of a forward-looking design. I'd much
rather see what we can mooch off PEP 3148 (Futures).

> 4+ adapting the stdlib tools to using these new things

We at least need to have an idea for how this could be done. We're
talking serious rewrites of many of our most fundamental existing
synchronous protocol libraries (e.g. httplib, email, possibly even
io.TextWrapper), most of which have had only scant updates even
through the Python 3 transition apart from complications to deal with
the bytes/str dichotomy.

> Re: forward path for existing asyncore code. I don't remember this being
> raised as an issue. If anything, it was mentioned in passing, and I think
> the answer to it was something to the tune of "asyncore's API is broken,
> fixing it is more important than backwards compat". Essentially I agree with
> Guido that the important part is an upgrade path to a good third-party
> library, which is the part about asyncore that REALLY sucks right now.

I have the feeling that the main reason asyncore sucks is that it
requires you to subclass its Dispatcher class, which has a rather
treacherous interface.

> Regardless, an API upgrade is probably a good idea. I'm not sure if it
> should go in the first PEP: given the separation I've outlined above (which
> may be too spread out...), there's no obvious place to put it besides it
> being a new PEP.

Aren't all your proposals API upgrades?

> Re base reactor interface: drawing maximally from the lessons learned in
> twisted, I think IReactorCore (start, stop, etc), IReactorTime (call later,
> etc), asynchronous-looking name lookup, fd handling are the important parts.

That actually sounds more concrete than I'd like a reactor interface
to be. In the App Engine world, there is a definite need for a
reactor, but it cannot talk about file descriptors at all -- all I/O
is defined in terms of RPC operations which have their own (several
layers of) async management but still need to be plugged in to user
code that might want to benefit from other reactor functionality such
as scheduling and placing a call at a certain moment in the future.

> call_every can be implemented in terms of call_later on a separate object,
> so I think it should be (eg twisted.internet.task.LoopingCall). One thing
> that is apparently forgotten about is event loop integration. The prime way
> of having two event loops cooperate is *NOT* "run both in parallel", it's
> "have one call the other". Even though not all loops support this, I think
> it's important to get this as part of the interface (raise an exception for
> all I care if it doesn't work).

This is definitely one of the things we ought to get right. My own
thoughts are slightly (perhaps only cosmetically) different again:
ideally each event loop would have a primitive operation to tell it to
run for a little while, and then some other code could tie several
event loops together.

Possibly the primitive operation would be something like "block until
either you've got one event ready, or until a certain time (possibly
0) has passed without any events, and then give us the events that are
ready and a lower bound for when you might have more work to do" -- or
maybe instead of returning the event(s) it could just call the
associated callback (it might have to if it is part of a GUI library
that has callbacks written in C/C++ for certain events like screen
refreshes).

Anyway, it would be good to have input from representatives from Wx,
Qt, Twisted and Tornado to ensure that the *functionality* required is
all there (never mind the exact signatures of the APIs needed to
provide all that functionality).

-- 
--Guido van Rossum (python.org/~guido)