[Python-Dev] PEP 380 (yield from a subgenerator) comments

[P.J. Eby]

At 09:24 PM 3/25/2009 -0700, Guido van Rossum wrote:
>ISTR that the motivation for adding new syntax is that the best you
>can do using a trampoline library is still pretty cumbersome to use
>when you have to write a lot of tasks and subtasks, and when using
>tasks is just a tool for getting things done rather than an end goal
>in itself. I agree that the motivation and the comparison should be
>added to the PEP (perhaps moving the trampoline sample
>*implementation* to a reference or an appendix, since it is only the
>appearance of the trampoline-*using* code that matters).

In particular, it should explain why these choices are so costly as 
to justify new syntax and a complex implementation:

* decorator clearly identifying intent to create a task vs. no 
indication of task-ness
* "yield Return(value)" vs. "return value"
* "result = yield subgenerator()" vs. "result = yield from subgenerator()"

Especially since the first two differences arguably make the 
non-"yield from" code clearer, and the third makes it more compact.


>with judicious use of decorators and helper classes you can get a
>reasonable approximation, and I think Twisted uses something like
>this, so it's not just theory. I think the best you can do without new
>syntax though is still pretty cumbersome and brittle, which is why I
>have encouraged your PEP.

On the "cumbersome" side, there are only three differences, as I've 
shown above... and one of them uses less syntax than the PEP.

I'm not sure what you mean by brittle.  Twisted and PEAK have both 
had generator-based tasks for ages, and have been used in production 
for years, even before generators had send()/throw() -- the addition 
of yield expressions made it possible to get rid of the one previous 
brittleness, where you needed to do things like:

       yield subgenerator(); result = resume()

in order to pass values or exceptions in.  Since send()/throw() was 
added, the need to call a function after each yield was eliminated, 
at least from PEAK and my newer Trellis library; haven't looked at 
Twisted's tasks in a while.  I believe there are other 
generator-based task libraries available on PyPI, but can't comment 
on their robustness.

(The existence of throw(), by the way, makes it possible to produce 
tracebacks that look *exactly* as if you had called a series of 
functions, rather than iterating over a bunch of generators.  The 
sample code I gave should do this correctly, as it was created by 
yanking out working, tested code from my Trellis library, and cutting 
out all the Trellis-specific bits.)

If someone can find any semantic differences between the code I 
posted and the yield-from proposal (apart from the absence of the 
"for x in y: yield x" part of the functionality), I'd like to know 
about it...  and it should go in the PEP.