[Python-ideas] Yielding through context managers

[Nick Coghlan]

On Mon, Jan 7, 2013 at 2:24 AM, Guido van Rossum <guido at python.org> wrote:
> On Sunday, January 6, 2013, Nick Coghlan wrote:
>>
>> On Sun, Jan 6, 2013 at 8:20 PM, Laurens Van Houtven <_ at lvh.cc> wrote:
>> > Hi Nick,
>> >
>> >
>> > When you say "high latency" (in __exit__), what does "high" mean? Is
>> > that
>> > order of magnitude what __exit__ usually means now, or network IO
>> > included?
>> >
>> > (Use case: distributed locking and remotely stored locks: it doesn't
>> > take a
>> > long time on network scales, but it can take a long time on CPU scales.)
>>
>> The status quo can only be made to work for in-memory locks. If the
>> release step involves network access, then it's closer to the
>> "database transaction" use case, because the __exit__ method may need
>> to block.
>
> But you don't need to wait for the release. You can do that asynchronously.

Ah, true, I hadn't thought of that. So yes, any case where the
__exit__ method can be "fire-and-forget" is also straightforward to
implement with just PEP 3156. That takes us back to things like
database transactions being the only ones where

> Also, have you given the implementation of your 'yielding' proposal any
> thought yet?

Not in depth. Off the top of my head, I'd suggest:
  - make "yielding" a new kind of node in the grammar (so you can't
write "yielding expr" in arbitrary locations, but only in those that
are marked as allowing it)
  - flag for loops and with statements as accepting these nodes as
iterables and context managers respectively
  - create a new Yielding AST node (with a single Expr node as the child)
  - emit different bytecode in the affected compound statements based
on whether the relevant subnode is an ordinary expression (thus
invoking the special methods as "obj.__method__()") or a yielding one
(thus invoking the special methods as "yield from obj.__method__()").

I'm not seeing any obvious holes in that strategy, but I haven't
looked closely at the compiler code in a while, so there may be
limitations I haven't accounted for.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia