outsmarting context managers with coroutines

[Ian Kelly]

On Sun, Dec 29, 2013 at 7:44 AM, Burak Arslan
<burak.arslan at arskom.com.tr> wrote:
> On 12/29/13 07:06, Ian Kelly wrote:
>> On Sat, Dec 28, 2013 at 5:35 PM, Burak Arslan
>> <burak.arslan at arskom.com.tr> wrote:
>>> On 12/29/13 00:13, Burak Arslan wrote:
>>>> Hi,
>>>>
>>>> Have a look at the following code snippets:
>>>> https://gist.github.com/plq/8164035
>>>>
>>>> Observations:
>>>>
>>>> output2: I can break out of outer context without closing the inner one
>>>> in Python 2
>>>> output3: Breaking out of outer context closes the inner one, but the
>>>> closing order is wrong.
>>>> output3-yf: With yield from, the closing order is fine but yield returns
>>>> None before throwing.
>>> It doesn't, my mistake. Python 3 yield from case does the right thing, I
>>> updated the gist. The other two cases still seem weird to me though. I
>>> also added a possible fix for python 2 behaviour in a separate script,
>>> though I'm not sure that the best way of implementing poor man's yield from.
>> I don't see any problems here.  The context managers in question are
>> created in separate coroutines and stored on separate stacks, so there
>> is no "inner" and "outer" context in the thread that you posted.  I
>> don't believe that they are guaranteed to be called in any particular
>> order in this case, nor do I think they should be.
>
> First, Python 2 and Python 3 are doing two separate things here: Python
> 2 doesn't destroy an orphaned generator and waits until the end of the
> execution. The point of having raw_input at the end is to illustrate
> this. I'm tempted to call this a memory leak bug, especially after
> seeing that Python 3 doesn't behave the same way.

Ah, you may be right.  I'm not sure what's going on with Python 2
here, and all my attempts to collect the inaccessible generator have
failed.  The only times it seems to clean up are when Python exits
and, strangely, when dropping to an interactive shell using the -i
command line option.  It also cleans up properly if the first
generator explicitly dels its reference to the second before exiting.
This doesn't have anything to do with the context manager though, as I
see the same behavior without it.

> As for the destruction order, I don't agree that destruction order of
> contexts should be arbitrary. Triggering the destruction of a suspended
> stack should first make sure that any allocated objects get destroyed
> *before* destroying the parent object. But then, I can think of all
> sorts of reasons why this guarantee could be tricky to implement, so I
> can live with this fact if it's properly documented. We should just use
> 'yield from' anyway.

You generally get this behavior when objects are deleted by the
reference counting mechanism, but that is an implementation detail of
CPython.  Since different implementations use different collection
schemes, there is no language guarantee of when or in what order
finalizers will be called, and even in CPython with the new PEP 442,
there is no guarantee of what order finalizers will be called when
garbage collecting reference cycles.

>> For example, the first generator could yield the second generator back
>> to its caller and then exit, in which case the second generator would
>> still be active while the context manager in the first generator would
>> already have done its clean-up.
>
> Sure, if you pass the inner generator back to the caller of the outer
> one, the inner one should survive. The refcount of the inner is not zero
> yet. That's doesn't have much to do with what I'm trying to illustrate
> here though.

The point I'm trying to make is that either context has the potential
to long outlive the other one, so you can't make any guarantee that
they will be exited in LIFO order.