[Python-Dev] PEP 492: What is the real goal?

[Paul Moore]

On 29 April 2015 at 20:19, Paul Moore <p.f.moore at gmail.com> wrote:
> However, just to make my point precise, here's a more or less direct
> translation of the Wikipedia code into Python. It doesn't actually
> work, because getting the right combinations of yield and send stuff
> is confusing to me. Specifically, I suspect that "yield
> produce.send(None)" isn't the right way to translate "yield to
> produce". But it gives the idea.

Hmm, when I try to fix this "minor" (as I thought!) issue with my
code, I discover it's more fundamental. The error I get is

Traceback (most recent call last):
  File ".\coro.py", line 28, in <module>
    next(produce)
  File ".\coro.py", line 13, in produce
    yield consume.send(None)
  File ".\coro.py", line 23, in consume
    yield produce.send(None)
ValueError: generator already executing

What I now realise that means is that you cannot have producer send to
consumer which then sends back to producer. That's what the "generator
already executing" message means.

This is fundamentally different from the "traditional" use of
coroutines as described in the Wikipedia article, and as I thought was
implemented in Python. The Wikipedia example allows two coroutines to
freely yield between each other. Python, on the other hand, does not
support this - it requires the mediation of some form of "trampoline"
controller (or event loop, in asyncio terms) to redirect control. [1]

This limitation of Python's coroutines is not mentioned anywhere in
PEP 342, and that's probably why I never really understood Python
coroutines properly, as my mental model didn't match the
implementation.

Given that any non-trivial use of coroutines in Python requires an
event loop / trampoline, I begin to understand the logic behind
asyncio and this PEP a little better. I'm a long way behind in
understanding the details, but at least I'm no longer completely
baffled.

Somewhere, there should be an explanation of the difference between
Python's coroutines and Wikipedia's - I can't be the only person to be
confused like this. But I don't think there's any docs covering
"coroutines in Python" outside of PEP 342 - the docs just cover the
components (the send and throw methods, the yield expression, etc).
Maybe it could be covered in the send documentation (as that's what
gives the "generator already executing" error. I'll try to work up a
doc patch. Actually, looking at the docs, I can't even *find* where
the behaviour of the send method is defined - can someone point me in
the right direction?

Paul

[1] It's sort of similar to how Python doesn't do tail call
elimination. Symmetric yields rely on stack frames that are no longer
needed being discarded if they are to avoid unlimited recursion, so to
have symmetric yields, Python would need a form of tail call ("tail
yield", I guess :-)) elimination.