[Python-ideas] With clauses for generator expressions

[Phil Connell]

On Wed, Nov 14, 2012 at 07:44:44PM -0800, Andrew Barnert wrote:
> First, I realize that people regularly propose with expressions. This is not the 
> same thing.
> 
> The problem with the with statement is not that it can't be postfixed 
> perl-style, or used in expressions. The problem is that it can't be used with 
> generator expressions.
> 
> Here's the suggestion:
> 
>     upperlines = (lines.upper() for line in file with open('foo', 'r') as file)

While this looks very clean, how do you propose the following should be written
as a generator expression?

def foo():
    with open('foo') as f:
        for line in f:
            if 'bar' in line:
                yield line


An obvious suggestion is as follows, but I'm not totally convinced about the
out-of-order with, for and if clauses (compared with the equivalent generator)

(line
 for line in f
 if bar in 'line'
 with open('foo') as f)

Cheers,
Phil

> 
> This would be equivalent to:
> 
>     def foo():
>         with open('foo', 'r') as file:
>             for line in file:
>                 yield line.upper()
>     upperlines = foo()
> 
> The motivation is that there is no way to write this properly using a with 
> statement and a generator expression—in fact, the only way to get this right is 
> with the generator function above. And almost nobody ever gets it right, even 
> when you push them in the right direction (although occasionally they write a 
> complex class that has the same effect).
> 
> That's why we still have tons of code like this lying around:
> 
>     upperlines = (lines.upper() for line in open('foo', 'r'))
> 
> Everyone knows that this only works with CPython, and isn't even quite right 
> there, and yet people write it anyway, because there's no good alternative.
> 
> The with clause is inherently part of the generator expression, because the 
> scope has to be dynamic. The file has to be closed when iteration finishes, not 
> when creating the generator finishes (or when the generator is cleaned up—which 
> is closer, but still wrong).
> 
> That's why a general-purpose "with expression" wouldn't actually help here; in 
> fact, it would just make generator expressions with with clauses harder to 
> parse. A with expression would have to be statically scoped to be general.
> 
> For more details, see this:
> 
> http://stupidpythonideas.blogspot.com/2012/11/with-clauses-for-generator-expressions.html
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas