[Tutor] Yielding from a with block

[Steven D'Aprano]

On Thu, May 28, 2015 at 10:46:26AM +0100, Oscar Benjamin wrote:

> I'm sure you understand the fundamental difference between calling a
> function and yielding inside a with statement: when calling a function
> the new frame is *appended* to the call stack keeping the current
> frame and all of its exception traps and context managers in place
> ready for unwinding. When yielding the current frame is *removed* from
> the call stack (along with its exception traps and context managers).

Implementation details.


> When a with block is used without a yield it is not possible (barring
> unrecoverable failure of the runtime itself) to leave the block
> without calling its finaliser. 

Yes.


> It doesn't matter if we call a function
> that in turn calls a generator. 

This sentence is ambiguous. Do you mean "calls a generator function", or 
do you mean "call [next() on] a generator object"?

I don't suppose it really matters though.


> As long as there is no yield inside
> the with block in the current frame then the finalisation guarantee is
> maintained. When using yield we don't have this guarantee and in fact
> there are common non-exceptional cases (break/return) where the
> undesirable occurs.

Er, no. You still have the finalisation guarantee. You just have to 
understand what the guarantee actually is. It does *not* refer to 
pausing the generator to pass control back to the caller. That would 
make yield inside a with block absolutely useless.

The guarantee is that when you *exit* (not pause) the with block, then 
and only then will the context manager's __exit__ method run. That's no 
different from the non-generator use of a CM.

If you think that guarantee is not made, you have to either demonstrate 
a counter-example where exiting the with block fails to run the __exit__ 
method, or point us to official documentation supporting your position.

Otherwise I stand by my earlier position that you are misinterpreting 
what it means to exit a with block. Pausing it to yield is not an exit.

I did an experiment, where I tried to break the finalisation 
guarantee using break, return and raise:

class CM:
    def __enter__(self):
        return self
    def __exit__(self, *args):
        print("exiting")

def test(n):
    for i in range(1):
        with CM():
            if n == "break": break
            if n == "return": return
            if n == "raise": raise RuntimeError
            yield 1



Falling out the bottom of the generator finalises correctly. So do 
break, return and raise.

it = test("")
x = next(it)
next(it, None)  # prints "exiting"

it = test("break")
next(it, None)  # prints "exiting"

it = test("return")
next(it, None)  # prints "exiting"

it = test("raise")
try: next(it)
except: pass  # prints "exiting"



Under what circumstances can execution leave the with block without the 
finalisation method __exit__ running?



-- 
Steve