[Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory

[Stefan Behnel]

Nick Coghlan wrote:
> Stefan Behnel wrote:
>> This is also an important issue for other Python implementations. Cython
>> simply transforms comprehensions into the equivalent for-loop, so when we
>> implement PEP 342 in Cython, we will have to find a way to emulate
>> CPython's behaviour here (unless we decide to stick with Py2.x sematics,
>> which would not be my preferred solution).
> 
> How do you do that without leaking the iteration variable into the
> current namespace?

We currently have 2.x sematics for comprehensions anyway, but the
(long-standing) idea is to move comprehensions into their own scope (not a
function, just a new type of scope), so that all names defined inside the
expressions end up inside of the inner scope. This is completely orthogonal
to the loop transformation itself, though, which would simply happen inside
of the inner scope.

However, having to emulate the other Py3 semantics for comprehensions that
this thread is about, would pretty much kill such a simple solution.


> The translation of your example isn't quite as Alexandre describes it -
> we do at least avoid the overhead of creating a generator function in
> the list comprehension case. It's more like:
> 
>     while True:
>         def f():
>             result = []
>             for i in range(chunk_size):
>                 result.append((yield))
>             return result
>         target.send(f())

So the problem is that f(), i.e. the function-wrapped comprehension itself,
swallows the "(yield)" expression (which redundantly makes it a generator).
That means that the outer function in my example, which was

	def chunker(chunk_size, target):
	    while True:
	        target.send([ (yield) for i in range(chunk_size) ])

doesn't become a generator itself, so the above simply ends up as an
infinite loop.

IMHO, that's pretty far from obvious when you look at the code.

Also, the target receives a "generator object <listcomp>" instead of a
list. That sounds weird.


> It seems to me that CPython should be raising a SyntaxError for yield
> expressions inside comprehensions (in line with the "no returning values
> other than None from generator functions" rule), and probably for
> generator expressions as well.

Yes, that's what I was suggesting. Disallowing it in genexps is a more open
question, though. I wouldn't mind being able to send() values into a
generator expression, or to throw() exceptions during their execution.

Anyway, I have no idea about a use case, so it might just as well be
disallowed for symmetry reasons.

Stefan