[Tutor] Yielding from a with block

[Oscar Benjamin]

On 29 May 2015 at 14:29, Steven D'Aprano <steve at pearwood.info> wrote:
> On Thu, May 28, 2015 at 10:16:00AM +0200, Peter Otten wrote:
>
>> Even if you limit yourself to CPython there is another effect: the order of
>> execution may not meet one's expectations/requirements:
>
> [snip example]
>
> That's an interesting example, and I can't tell if that's a
> problem with your (and my) expectations, or a bug in the context
> manager implementation.

I think that this behaviour is consistent with the generally
unspecified language-level behaviour of deallocation and __del__.
Consider the language specification for this version of Python (and
note the last sentence in particular):

"""
object.__del__(self)

Called when the instance is about to be destroyed. This is also called
a destructor. If a base class has a __del__() method, the derived
class’s __del__() method, if any, must explicitly call it to ensure
proper deletion of the base class part of the instance. Note that it
is possible (though not recommended!) for the __del__() method to
postpone destruction of the instance by creating a new reference to
it. It may then be called at a later time when this new reference is
deleted. It is not guaranteed that __del__() methods are called for
objects that still exist when the interpreter exits.
"""
https://docs.python.org/2/reference/datamodel.html#object.__del__

> PEP 343 clearly warns that the finally clause may not run immediately:
>
>     Note that we're not guaranteeing that the finally-clause is
>     executed immediately after the generator object becomes unused
>
> https://www.python.org/dev/peps/pep-0343/
>

The rest of that sentence says "..., even though this is how it will
work in CPython." To me that clearly and deliberately permits other
implementations to behave differently.

> but the documentation for the with-statement suggests strongly that the
> __exit__ method will run immediately after the block is exited, before
> any additional code (including before any exception is raised. E.g.:
>
>     5. The suite is executed.
>
>     6. The context manager’s __exit__() method is invoked. If an
>        exception caused the suite to be exited, its type, value, and
>        traceback are passed as arguments to __exit__(). Otherwise,
>        three None arguments are supplied.
>
> https://docs.python.org/2/reference/compound_stmts.html#the-with-statement

But the with statement is not "exited" as the suite is unfinished. The
generator has suspended itself and removed itself from the call-stack
so that it is no longer permitted to catch exceptions etc. Also note
that the removal from the call stack is not an implementation detail.
The interpreter can implement the call stack how it likes but it must
be semantically equivalent to a call stack with exception unwinding in
order to meet the general definition of the language.

>> PS: I'm still looking for a fairly elegant rewrite of the problematic
>>
>> def lines(files):
>>     for file in files:
>>         with open(files) as f:
>>             yield from f
>>
>> (see Oscar's comment in
>> <https://mail.python.org/pipermail/tutor/2015-May/105448.html>)
>
> Oscar's comment includes a syntax error, which makes it hard to run his
> code:
>
>         print("__exit__ called")__del__.
>
> I can't even begin to guess what that is supposed to be, and reading the
> next few messages in the thread doesn't enlighten.

Some kind of editing error. It should be:

         print("__exit__ called")


--
Oscar