[Patches] [ python-Patches-1548388 ] set comprehensions

SourceForge.net noreply at sourceforge.net
Wed Mar 7 10:23:00 CET 2007 

Patches item #1548388, was opened at 2006-08-29 08:33
Message generated for change (Comment added) made by gbrandl
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1548388&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core (C code)
Group: Python 3000
>Status: Closed
>Resolution: Out of Date
Priority: 5
Private: No
Submitted By: Georg Brandl (gbrandl)
Assigned to: Georg Brandl (gbrandl)
Summary: set comprehensions

Initial Comment:
This is a big one:

* cleanup grammar; unifies listcomp/genexp grammar
which means that [x for x in 1, 2] is no longer valid

* cleanup comprehension compiling code (unifies all AST
code for the three comprehensions and most of the
compile.c code)

* add set comprehensions

This patch modifies list comprehensions to be
implemented more like generator expressions: in a
separate function, which means that the loop variables
will not leak any more.

----------------------------------------------------------------------

>Comment By: Georg Brandl (gbrandl)
Date: 2007-03-07 09:23

Message:
Logged In: YES 
user_id=849994
Originator: YES

This patch is now superseded by #1660500.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2007-01-04 04:57

Message:
Logged In: YES 
user_id=6380
Originator: NO

There was some discussion on the py3k list about Raymond's suggestion. 
Are you thinking of doing that?  I'd really like to see the syntactic
changes and additions from this patch, but I agree that for list/set comps
we can do without the extra stack frame.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2006-09-08 17:13

Message:
Logged In: YES 
user_id=80475

Genexps necessarily need a separate stack frame to achieve
saved execution state (including the instruction pointer and
local variable).  Also, it was simplest to implement genexps
in terms of the existing and proven code for regular generators.

For list and set comps, I think you can take a simpler
approach and just rename the inner loop variable to
something invisible.  That will make it faster, make the
disassemby readable, and make it easier to follow in pdb. 
Also, we get to capitalize on proven code -- they only
difference is that the induction variable won't be visible
to surrounding code.

Since what you have works, I would say just check it in;
however, it would probably never get touched again and an
early, arbitrary design choice would get set in stone.  My
bet is that the renaming approach will result in a much
simpler patch.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2006-09-06 15:36

Message:
Logged In: YES 
user_id=6380

I always assumed that the genexps *require* being a function
because that's the only way to create a generator.  But that
argument doesn't apply to listcomps.

That's about all I know of the implementation of these.. :-(

Have you asked python-dev?

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-09-06 07:03

Message:
Logged In: YES 
user_id=849994

It is complete, it works and it does not leak the loop
variable(s).

The question is whether it is okay for listcomps and
setcomps to be in their own anonymous function, which slows
listcomps down compared to the 2.x branch.

I don't know why the function approach was taken for
genexps, but I suspect it was because who implemented it
then saw this as the best way to hide the loop variable.

Perhaps somebody else more familiar with the internals and
the previous discussions can look over it.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2006-09-06 06:48

Message:
Logged In: YES 
user_id=6380

Do you think this is ready to be checked in, or are you
still working on it?

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-09-01 09:38

Message:
Logged In: YES 
user_id=849994

Since you can put anything usable as an assignment target
after the "for" of a listcomp, just renaming might be
complicated.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2006-08-31 23:40

Message:
Logged In: YES 
user_id=6380

+1.

Would this cause problems for abominations like this though?

>>> a=[1]
>>> list(tuple(a) for a[0] in "abc")
[('a',), ('b',), ('c',)]
>>> a
['c']
>>>


----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2006-08-31 23:15

Message:
Logged In: YES 
user_id=80475

Would it be an oversimplfication for list and set comps to
keep everything in one code block and just hide the list
loop variables by renaming them:   x -->  __[x]

That approach would only require a minimal patch, and it
would make for a cleaner disassembly.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-08-31 19:55

Message:
Logged In: YES 
user_id=849994

Attaching slightly revised patch and bytecode comparison.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2006-08-29 22:30

Message:
Logged In: YES 
user_id=80475

Can you post a before and disassembly of some list and set
comprehensions.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-08-29 19:09

Message:
Logged In: YES 
user_id=849994

test_compiler and test_transformer fail because the compiler
package hasn't been updated yet.

test_dis fails because list comprehensions now generate
different bytecode.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2006-08-29 17:59

Message:
Logged In: YES 
user_id=6380

Nice!

I see failures in 4 tests:

    test_compiler test_dis test_transformer test_univnewlines

test_univnewlines is trivial (it's deleting a variable
leaked out of a list comprehension); haven't looked at the
rest in detail

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-08-29 08:34

Message:
Logged In: YES 
user_id=849994

The previously attached patch contains only the important
files. The FULL patch (attached now) also contains syntax
fixes in python files so that the test suite is mostly passing.

Note that the compiler package isn't ready yet.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1548388&group_id=5470

More information about the Patches mailing list