[Python-Dev] Coroutines and PEP 380

[Matt Joiner]

On Fri, Jan 20, 2012 at 8:41 AM, Greg <greg.ewing at canterbury.ac.nz> wrote:
> Glyph wrote:
>>
>> [Guido] mentions the point that coroutines that can implicitly switch out
>> from under you have the same non-deterministic property as threads: you
>> don't know where you're going to need a lock or lock-like construct to
>> update any variables, so you need to think about concurrency more deeply
>> than if you could explicitly always see a 'yield'.
>
>
> I'm not convinced that being able to see 'yield's will help
> all that much. In any system that makes substantial use of
> generator-based coroutines, you're going to see 'yield from's
> all over the place, from the lowest to the highest levels.
> But that doesn't mean you need a correspondingly large
> number of locks. You can't look at a 'yield' and conclude
> that you need a lock there or tell what needs to be locked.
>
> There's no substitute for deep thought where any kind of
> theading is involved, IMO.
>
> --
> Greg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com

I wasn't aware that Guido had brought this up, and I believe what he
says to be true. Preemptive coroutines, are just a hack around the
GIL, and reduce OS overheads. It's the explicit nature of the enhanced
generators that is their greatest value.

FWIW, I wrote a Python 3 compatible equivalent to gevent (also
greenlet based, and also very similar to Brett's et al coroutine
proposal), which didn't really solve the concurrency problems I hoped.
There were no guarantees whether functions would "switch out", so all
the locking and threading issues simply reemerged, albeit with also
needing to have all calls non-blocking, losing compatibility with any
routine that didn't make use of nonblocking calls and/or expose it's
"yield" in the correct way, but reducing GIL contention. Overall not
worth it.

In short, implicit coroutines are just a GIL work around, that break
compatibility for little gain.

Thanks Glyph for those links.