outsmarting context managers with coroutines

[Burak Arslan]

On 12/29/13 07:06, Ian Kelly wrote:
> On Sat, Dec 28, 2013 at 5:35 PM, Burak Arslan
> <burak.arslan at arskom.com.tr> wrote:
>> On 12/29/13 00:13, Burak Arslan wrote:
>>> Hi,
>>>
>>> Have a look at the following code snippets:
>>> https://gist.github.com/plq/8164035
>>>
>>> Observations:
>>>
>>> output2: I can break out of outer context without closing the inner one
>>> in Python 2
>>> output3: Breaking out of outer context closes the inner one, but the
>>> closing order is wrong.
>>> output3-yf: With yield from, the closing order is fine but yield returns
>>> None before throwing.
>> It doesn't, my mistake. Python 3 yield from case does the right thing, I
>> updated the gist. The other two cases still seem weird to me though. I
>> also added a possible fix for python 2 behaviour in a separate script,
>> though I'm not sure that the best way of implementing poor man's yield from.
> I don't see any problems here.  The context managers in question are
> created in separate coroutines and stored on separate stacks, so there
> is no "inner" and "outer" context in the thread that you posted.  I
> don't believe that they are guaranteed to be called in any particular
> order in this case, nor do I think they should be.

First, Python 2 and Python 3 are doing two separate things here: Python
2 doesn't destroy an orphaned generator and waits until the end of the
execution. The point of having raw_input at the end is to illustrate
this. I'm tempted to call this a memory leak bug, especially after
seeing that Python 3 doesn't behave the same way.

As for the destruction order, I don't agree that destruction order of
contexts should be arbitrary. Triggering the destruction of a suspended
stack should first make sure that any allocated objects get destroyed
*before* destroying the parent object. But then, I can think of all
sorts of reasons why this guarantee could be tricky to implement, so I
can live with this fact if it's properly documented. We should just use
'yield from' anyway.


> For example, the first generator could yield the second generator back
> to its caller and then exit, in which case the second generator would
> still be active while the context manager in the first generator would
> already have done its clean-up.

Sure, if you pass the inner generator back to the caller of the outer
one, the inner one should survive. The refcount of the inner is not zero
yet. That's doesn't have much to do with what I'm trying to illustrate
here though.

Best,
Burak