yield in try/finally case

[Peter Otten]

刘琦帆 wrote:

> 在 2016年3月3日星期四 UTC+8下午8:14:29，Oscar Benjamin写道：
>> On 3 March 2016 at 11:52, 刘琦帆 <lqf.txx at gmail.com> wrote:
>> >
>> > "A yield statement is not allowed in the try clause of a try/finally
>> > construct.  The difficulty is that there's no guarantee the generator
>> > will ever be resumed, hence no guarantee that the finally block will
>> > ever get executed; that's too much a violation of finally's purpose to
>> > bear." from https://www.python.org/dev/peps/pep-0255/
>> >
>> > But, meanwhile, the code showed on that page use yield in a try/finally
>> > case. It really puzzles me. Is there anything wrong?
>> 
>> I think what it means is that you can put a yield in the finally block
>> but not the try block so:
>> 
>> # Not allowed
>> def  f():
>>     try:
>>         yield 1
>>     finally:
>>         pass
>> 
>> # Allowed
>> def f():
>>     try:
>>         pass
>>     finally:
>>         yield 1
>> 
>> However that information is out of date. The restriction was removed
>> in some later Python version. Actually the construct is quite common
>> when using generator functions to implement context managers:
>> 
>> @contextlib.contextmanager
>> def replace_stdin(newstdin):
>>     oldstdin = sys.stdin
>>     try:
>>         sys.stdin = newstdin
>>         yield
>>     finally:
>>         sys.stdin = oldstdin
>> 
>> Although the restriction was removed the problem itself still remains.
>> There's no guarantee that a finally block will execute if there is a
>> yield in the try block. The same happens if you use a context manager
>> around a yield statement: the __exit__ method is not guaranteed to be
>> called. One implication of this is that in the following code it is
>> not guaranteed that the file will be closed:
>> 
>> def upperfile(filename):
>>     with open(filename) as fin:
>>         for line in fin:
>>             yield line.upper()
>> 
>> --
>> Oscar
> 
> 
> It really nice of you to answer the question. But I am still confused with
> your last example, is there any case that the file with not be closed? I
> just run the code and no exception occur.

It doesn't happen easily, you have to defeat CPython's garbage collection. 
Consider the follwing script:

$ cat upper1.py
import sys

_open = open
files = []

def open(*args, **kw):
    """Use a modified open() which keeps track of opened files.

    This allows us to check whether the files are properly closed and
    also to defeat garbage collection.
    """
    f = _open(*args, **kw)
    files.append(f)
    return f


for filename in sys.argv[1:]:
    with open(filename) as f:
        for line in f:
            print(line.upper(), end="")
            break

assert all(f.closed for f in files)

$ echo -e 'foo\nbar\nbaz' > tmp1.txt
$ echo -e 'hams\nspam\njam' > tmp2.txt
$ python3 upper1.py *.txt
FOO
HAMS

As expected it prints the first lines of the files provided as commandline 
args, in upper case. Now let's refactor:

$ cat upper2.py
import sys

_open = open
files = []

def open(*args, **kw):
    """Use a modified open() which keeps track of opened files.

    This allows us to check whether the files are properly closed (and
    also defeats garbage collection).
    """
    f = _open(*args, **kw)
    files.append(f)
    return f

def upperfile(filename):
    with open(filename) as f:
        for line in f:
            yield line.upper()

for uf in map(upperfile, sys.argv[1:]):
    for line in uf:
        print(line, end="")
        break

assert all(f.closed for f in files)

The change looks harmless, we moved the with statement and the conversion 
into the generater. But when whe run it:

$ python3 upper2.py *.txt
FOO
HAMS
Traceback (most recent call last):
  File "upper2.py", line 26, in <module>
    assert all(f.closed for f in files)
AssertionError

This is because the last generator uf = upperfile(...) is not garbage-
collected and wasn't explicitly closed either. If you do care here's one 
possible fix:

from contextlib import closing
...
for uf in map(upperfile, sys.argv[1:]):
    with closing(uf):
        for line in uf:
            print(line, end="")
            break