From ncoghlan at gmail.com Wed Jun 3 08:35:35 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 03 Jun 2009 16:35:35 +1000 Subject: [Python-ideas] with statement vs. try...except...finally In-Reply-To: <5d1a32000905291118t4d1fa5b2saa899d89db53122d@mail.gmail.com> References: <5d1a32000905291118t4d1fa5b2saa899d89db53122d@mail.gmail.com> Message-ID: <4A2619B7.4040300@gmail.com> Gerald Britton wrote: > Fine so far, but what if I want to be more granular? e.g. with "try...except": > > try: > f = open('foo') > except IOError: > line = "can't open" > > try: > line = f.readline() > except IOError: > line = "can't read" > > try: > f.close() > except IOError: > line = "can't close" > > I can't see how to replace the above try-triplet with a "with" > encapsulation. The with statement is designed to simplify certain common use cases of the try statement. That simplification comes at the cost of reduced flexibility. That's OK though: when you need the extra fine-grained control then the original try statement is still around to help you out. Increasing the complexity of the with statement so that it can cover every conceivable try statement use case would defeat the whole point of adding the new statement in the first place. Cheers, Nick. P.S. You could always factor the above out into a simple function: def get_first_line(fname): try: f = open('foo') except IOError: line = "can't open" try: line = f.readline() except IOError: line = "can't read" try: f.close() except IOError: line = "can't close" return line line = get_first_line('foo') Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From g.brandl at gmx.net Wed Jun 3 09:11:36 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 03 Jun 2009 09:11:36 +0200 Subject: [Python-ideas] with statement vs. try...except...finally In-Reply-To: <4A2619B7.4040300@gmail.com> References: <5d1a32000905291118t4d1fa5b2saa899d89db53122d@mail.gmail.com> <4A2619B7.4040300@gmail.com> Message-ID: Nick Coghlan schrieb: > def get_first_line(fname): > try: > f = open('foo') > except IOError: > line = "can't open" > try: > line = f.readline() > except IOError: > line = "can't read" > try: > f.close() > except IOError: > line = "can't close" > return line > > line = get_first_line('foo') Anyway, doesn't it make more sense to report not what *operation* failed, but what the *failure* was (the specifics of the IOError)? Georg From ncoghlan at gmail.com Wed Jun 3 09:40:33 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 03 Jun 2009 17:40:33 +1000 Subject: [Python-ideas] with statement vs. try...except...finally In-Reply-To: References: <5d1a32000905291118t4d1fa5b2saa899d89db53122d@mail.gmail.com> <4A2619B7.4040300@gmail.com> Message-ID: <4A2628F1.4030007@gmail.com> Georg Brandl wrote: > Anyway, doesn't it make more sense to report not what *operation* > failed, but what the *failure* was (the specifics of the IOError)? It probably isn't a good idea to overanalyse the toy example ;) Hypocritically-yours, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From zac256 at gmail.com Wed Jun 10 02:13:16 2009 From: zac256 at gmail.com (Zac Burns) Date: Tue, 9 Jun 2009 17:13:16 -0700 Subject: [Python-ideas] os.listdir with current working directory as default In-Reply-To: References: <94bdd2610905221551k5e4ab70eue973ffdb2d84fd9f@mail.gmail.com> <94bdd2610905221619m6a4b7e79m4dcf2ce407fb301a@mail.gmail.com> <94bdd2610905221701v96a1b59h5aeada22bc0a092b@mail.gmail.com> <91ad5bf80905221709r23bcd712k6814886233ef48b5@mail.gmail.com> <94bdd2610905231708o1222551axf7ef449fc1329621@mail.gmail.com> Message-ID: <333edbe80906091713ma098b0bwd20491725c9cac71@mail.gmail.com> > You'd better be careful with it. The "current directory" is process-wide and so > modifying it in a thread will affect all other threads in your program. I'm not > saying all programs have this characteristic, but relying on the current > directory rather than explicit paths isn't always a good idea... > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > I agree with Antoine, but would go further to say that making it a default argument is an enablement - just making it easier for people to follow a bad pattern. This doesn't apply to all default arguments of course, just that the working directory is "considered harmful" and we probably shouldn't make changes that make it easier for people to use. -- Zachary Burns (407)590-4814 Aim - Zac256FL Production Engineer (Digital Overlord) Zindagi Games From lie.1296 at gmail.com Wed Jun 10 08:25:32 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Wed, 10 Jun 2009 16:25:32 +1000 Subject: [Python-ideas] os.listdir with current working directory as default In-Reply-To: <333edbe80906091713ma098b0bwd20491725c9cac71@mail.gmail.com> References: <94bdd2610905221551k5e4ab70eue973ffdb2d84fd9f@mail.gmail.com> <94bdd2610905221619m6a4b7e79m4dcf2ce407fb301a@mail.gmail.com> <94bdd2610905221701v96a1b59h5aeada22bc0a092b@mail.gmail.com> <91ad5bf80905221709r23bcd712k6814886233ef48b5@mail.gmail.com> <94bdd2610905231708o1222551axf7ef449fc1329621@mail.gmail.com> <333edbe80906091713ma098b0bwd20491725c9cac71@mail.gmail.com> Message-ID: Zac Burns wrote: >> You'd better be careful with it. The "current directory" is process-wide and so >> modifying it in a thread will affect all other threads in your program. I'm not >> saying all programs have this characteristic, but relying on the current >> directory rather than explicit paths isn't always a good idea... > > I agree with Antoine, but would go further to say that making it a > default argument is an enablement - just making it easier for people > to follow a bad pattern. > > This doesn't apply to all default arguments of course, just that the > working directory is "considered harmful" and we probably shouldn't > make changes that make it easier for people to use. When you open() files do you always use full path? Why don't we eliminate all modules and built-in function that uses relative path by default then? That way we will always be safe? Making open(), etc raise RelativePathError if using relative path. From denis.spir at free.fr Sat Jun 13 10:25:19 2009 From: denis.spir at free.fr (spir) Date: Sat, 13 Jun 2009 10:25:19 +0200 Subject: [Python-ideas] file(fn).read() Message-ID: <20090613102519.6bef9943@o> Hello, this post is half a question, half a proposal. Q: what happens to the filesystem file, and to the python file object below? text = file(fn).read() (used 'file' instead of 'open' in purpose) Note that there seemingly is no way to close or del the file object, nor even to check its 'closed' flag. I would expect that, in the case a file remains unnnamed, it is automatically closed and del-ed. If not, do you think it is sensible to ensure that? In this case, we have a both safe and compact idiom to read file content. I consider this good, because the concept of "read file content" is both a single and simple operation -- and rather common. Else, I would propose a pair of string methods (rather than 2 more builtin funcs) .filetext() & .filebytes(). In both cases, the target string is (supposed to be) a filename. Below an example. =================== class FString(str): ''' custom string implementing file content reading ''' TEXT_MODE = 'r' BYTES_MODE = 'rb' NO_FILE_MESSAGE = 'Cannot open file "%s".' def filetext(self): return self._filecontent(FString.TEXT_MODE) def filebytes(self): return self._filecontent(FString.BYTES_MODE) def _filecontent(self, mode): try: f = open(self, mode) except IOError,e: raise ValueError(FString.NO_FILE_MESSAGE % self) bytes = f.read() f.close() return bytes =================== >>> from FString import FString >>> fn = FString("test.txt") >>> fn.filetext() 'foo\nbar\n' >>> fn.filebytes() 'foo\nbar\n' >>> fn = FString("fool.txt") >>> fn.filetext() Traceback (most recent call last): File "", line 1, in File "FString.py", line 12, in filetext raise ValueError(NO_FILE_MESSAGE % fn) ValueError: Cannot open file "fool.txt". =================== I take the opportunity to ask whether there is a chance in the future to be able to customize builtin types. (In this case many proposals on this list will simply vanish.) Denis ------ la vita e estrany From pyideas at rebertia.com Sat Jun 13 12:44:33 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Sat, 13 Jun 2009 03:44:33 -0700 Subject: [Python-ideas] file(fn).read() In-Reply-To: <20090613102519.6bef9943@o> References: <20090613102519.6bef9943@o> Message-ID: <50697b2c0906130344o70dddab2oacf87fdf4913361f@mail.gmail.com> On Sat, Jun 13, 2009 at 1:25 AM, spir wrote: > Hello, > > this post is half a question, half a proposal. > > Q: what happens to the filesystem file, and to the python file object below? > > ? ?text = file(fn).read() > > (used 'file' instead of 'open' in purpose) > Note that there seemingly is no way to close or del the file object, nor even to check its 'closed' flag. > I would expect that, in the case a file remains unnnamed, it is automatically closed and del-ed. If not, do you think it is sensible to ensure that? Indeed, that is what happens. However, the language makes no guarantees as to exactly *when* it will be GC-ed and closed; there's nothing to ensure it will be closed in anything approaching a timely manner, which could cause problems in the case of long-running programs and/or programs that open many files. It is true that for CPython, the file object will happen to be GC-ed immediately, but this is mere implementation-specific behavior (due to refcounting being employed) and probably ought not to be relied upon; IronPython, for instance, does not behave the same way. > In this case, we have a both safe and compact idiom to read file content. I consider this good, because the concept of "read file content" is both a single and simple operation -- and rather common. Actually, we already have something for that. And it's only 1 extra line: with f as file(fn): text = f.read() #when execution reaches here, the file is guaranteed to be closed Though personally I think dealing with individual lines in a file is more common than slurping in an entire file's contents all at once. But I digress. > Else, I would propose a pair of string methods (rather than 2 more builtin funcs) .filetext() & .filebytes(). In both cases, the target string is (supposed to be) a filename. Probably unnecessary seeing as we have the above idiom. > I take the opportunity to ask whether there is a chance in the future to be able to customize builtin types. (In this case many proposals on this list will simply vanish.) I think you'll have to elaborate on what exactly you mean by "customizing builtin types" for anyone to answer your question fully. Assuming you mean "modify, replace, or extend in-place", I think the prospects of that are rather grim; Python is not Ruby or Lisp. Cheers, Chris -- http://blog.rebertia.com From denis.spir at free.fr Sat Jun 13 17:05:47 2009 From: denis.spir at free.fr (spir) Date: Sat, 13 Jun 2009 17:05:47 +0200 Subject: [Python-ideas] file(fn).read() In-Reply-To: <20090613102519.6bef9943@o> References: <20090613102519.6bef9943@o> Message-ID: <20090613170547.3c4a19a4@o> Le Sat, 13 Jun 2009 10:25:19 +0200, spir s'exprima ainsi: > Hello, > > this post is half a question, half a proposal. > > Q: what happens to the filesystem file, and to the python file object below? > > text = file(fn).read() > > (used 'file' instead of 'open' in purpose) > Note that there seemingly is no way to close or del the file object, nor > even to check its 'closed' flag. I would expect that, in the case a file > remains unnnamed, it is automatically closed and del-ed. If not, do you > think it is sensible to ensure that? Actually, I could have answered my own question, at least for the current CPython implementation: >>> file("test.txt").read() 'foo\nbar\n' >>> file("test.txt",'r') Seems to shows the file has been closed. Denis ------ la vita e estrany From Scott.Daniels at Acm.Org Sat Jun 13 19:57:05 2009 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sat, 13 Jun 2009 10:57:05 -0700 Subject: [Python-ideas] file(fn).read() In-Reply-To: <20090613170547.3c4a19a4@o> References: <20090613102519.6bef9943@o> <20090613170547.3c4a19a4@o> Message-ID: spir wrote: > .... Actually, I could have answered my own question, > at least for the current CPython implementation: .... Warning: experimentation only gets you to the behavior of a particular implementation on your machine at the time you run it. If you don't encounter a bug, it also gives you a _sample_ of _acceptable_ behavior. It cannot, however get you to the definition of the the behavior. For that you _must_ read the documentation. If you rely on experimentation, you will come to depend on a lot of things that might very well change. At best, your program will be much less portable, and at worst, the program will work perfectly until you demo or deliver your program. --Scott David Daniels Scott.Daniels at Acm.Org From josiah.carlson at gmail.com Sun Jun 14 20:26:16 2009 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Sun, 14 Jun 2009 11:26:16 -0700 Subject: [Python-ideas] file(fn).read() In-Reply-To: <20090613170547.3c4a19a4@o> References: <20090613102519.6bef9943@o> <20090613170547.3c4a19a4@o> Message-ID: On Sat, Jun 13, 2009 at 8:05 AM, spir wrote: > Le Sat, 13 Jun 2009 10:25:19 +0200, > spir s'exprima ainsi: > >> Hello, >> >> this post is half a question, half a proposal. >> >> Q: what happens to the filesystem file, and to the python file object below? >> >> ? ? text = file(fn).read() >> >> (used 'file' instead of 'open' in purpose) >> Note that there seemingly is no way to close or del the file object, nor >> even to check its 'closed' flag. I would expect that, in the case a file >> remains unnnamed, it is automatically closed and del-ed. If not, do you >> think it is sensible to ensure that? > > Actually, I could have answered my own question, at least for the current CPython implementation: > >>>> file("test.txt").read() > 'foo\nbar\n' >>>> file("test.txt",'r') > > > Seems to shows the file has been closed. How so? Reading a file isn't exclusive on any major Python platform. You can get file locking on various platforms by using platform-specific file locking calls, but that's not the default for Python. - Josiah From rdmurray at bitdance.com Mon Jun 15 04:20:15 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 15 Jun 2009 02:20:15 +0000 (UTC) Subject: [Python-ideas] file(fn).read() References: <20090613102519.6bef9943@o> Message-ID: spir wrote: > I take the opportunity to ask whether there is a chance in the future to be > able to customize builtin types. (In this case many proposals on this list > will simply vanish.) Assuming I understand your question correctly, then according to a recent posting on python-dev[1], builtin types not being modifiable is a policy decision. According to Guido, it is so that 3rd party modules cannot make incompatible changes to built-in types. So the answer to your question is most likely "no". --David [1] http://mail.python.org/pipermail/python-dev/2009-June/090044.html From cmjohnson.mailinglist at gmail.com Tue Jun 16 13:54:10 2009 From: cmjohnson.mailinglist at gmail.com (Carl Johnson) Date: Tue, 16 Jun 2009 01:54:10 -1000 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals Message-ID: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> I thought of this in the shower. It might not be a good idea, but I'd like to hear what other people think. On analogy with b"", r"", etc. we could introduce an empty set-literal and an odict-literal, and add a more explicit form to replace the existing set literal. s{} could be the empty set, o{} could be an empty odict, and we could leave {} alone as the form for dicts. So, an odict literal would look like o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), ('b', '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could (optionally?) have a little s{'a', 'c', 'b'} to make it more explicit that this is a set, not a dict. I suppose if we wanted to get crazy, we could also have a frozenset literal, fs{}? So what do people think? Is this too ugly to do? Does it confuse users who are used to C-style braces? Or is it a logical extension of the b"", r"", etc. system that could help make things follow EIBI better? -- Carl Johnson From ben+python at benfinney.id.au Tue Jun 16 15:55:40 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 16 Jun 2009 23:55:40 +1000 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> Message-ID: <87prd42sgz.fsf@benfinney.id.au> Carl Johnson writes: > we could introduce an empty set-literal and an odict-literal, and add > a more explicit form to replace the existing set literal. What do you mean by ?more explicit?? The existing set literal syntax is quite explicit. > s{} could be the empty set, o{} could be an empty odict, and we could > leave {} alone as the form for dicts. So, an odict literal would look > like o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), > ('b', '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could > (optionally?) have a little s{'a', 'c', 'b'} to make it more explicit > that this is a set, not a dict. I don't think that word ?explicit? means what you think it means. > So what do people think? Is this too ugly to do? You haven't really identified a problem that is solved by this. And yes, I think it's significantly uglier than the existing syntax. > Does it confuse users who are used to C-style braces? Or is it a > logical extension of the b"", r"", etc. system that could help make > things follow EIBI better? (Presuming you mean ?EIBTI?, for ?Explicit Is Better Than Implicit?.) I don't see anything implicit about the following forms: set() OrderedDict() frozenset() So I think you're trying to achieve something else other than ?explicit?, but I don't know what it is, nor what the problem is that needs addressing. -- \ ?Never express yourself more clearly than you are able to | `\ think.? ?Niels Bohr | _o__) | Ben Finney From gerald.britton at gmail.com Tue Jun 16 16:11:30 2009 From: gerald.britton at gmail.com (Gerald Britton) Date: Tue, 16 Jun 2009 10:11:30 -0400 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <87prd42sgz.fsf@benfinney.id.au> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> Message-ID: <5d1a32000906160711m76337108vd3bd036fc41ea089@mail.gmail.com> We surely need a real-world example of a problem this solves. Without something dramatically compelling, I can't imagine that your proposed syntax will have a ghost of a chance of being taken seriously. On Tue, Jun 16, 2009 at 9:55 AM, Ben Finney wrote: > Carl Johnson > writes: > >> we could introduce an empty set-literal and an odict-literal, and add >> a more explicit form to replace the existing set literal. > > What do you mean by ?more explicit?? The existing set literal syntax > is quite explicit. > >> s{} could be the empty set, o{} could be an empty odict, and we could >> leave {} alone as the form for dicts. So, an odict literal would look >> like o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), >> ('b', '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could >> (optionally?) have a little s{'a', 'c', 'b'} to make it more explicit >> that this is a set, not a dict. > > I don't think that word ?explicit? means what you think it means. > >> So what do people think? Is this too ugly to do? > > You haven't really identified a problem that is solved by this. And yes, > I think it's significantly uglier than the existing syntax. > >> Does it confuse users who are used to C-style braces? Or is it a >> logical extension of the b"", r"", etc. system that could help make >> things follow EIBI better? > > (Presuming you mean ?EIBTI?, for ?Explicit Is Better Than Implicit?.) > > I don't see anything implicit about the following forms: > > ? ?set() > ? ?OrderedDict() > ? ?frozenset() > > So I think you're trying to achieve something else other than > ?explicit?, but I don't know what it is, nor what the problem is that > needs addressing. > > -- > ?\ ? ? ? ? ? ?Never express yourself more clearly than you are able to | > ?`\ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? think.? ?Niels Bohr | > _o__) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?| > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Gerald Britton From rrr at ronadam.com Tue Jun 16 16:27:46 2009 From: rrr at ronadam.com (Ron Adam) Date: Tue, 16 Jun 2009 09:27:46 -0500 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <87prd42sgz.fsf@benfinney.id.au> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> Message-ID: <4A37ABE2.6070302@ronadam.com> Ben Finney wrote: > Carl Johnson > writes: > >> we could introduce an empty set-literal and an odict-literal, and add >> a more explicit form to replace the existing set literal. > > What do you mean by ?more explicit?? The existing set literal syntax > is quite explicit. > >> s{} could be the empty set, o{} could be an empty odict, and we could >> leave {} alone as the form for dicts. So, an odict literal would look >> like o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), >> ('b', '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could >> (optionally?) have a little s{'a', 'c', 'b'} to make it more explicit >> that this is a set, not a dict. > > I don't think that word ?explicit? means what you think it means. I think Carl is thinking that it is more concise and easier to read. When you read OrderedDict([('a', '1'),('b', '2'), ('c', '3')]), your mind is parsing *Ordered Dictionary from list of these three tuples*. Whebn you read *o{'a':'1', 'b':'2', 'c':'3'}*, your mind does not have the extra noise of converting the argument list of the constructor. I think it really depends on how often ordered dictionaries literals will be used weather or not it is worth doing. It also uses about 30% to 50% less keystrokes. Ron From python at mrabarnett.plus.com Tue Jun 16 19:46:21 2009 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 16 Jun 2009 18:46:21 +0100 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <4A37ABE2.6070302@ronadam.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> Message-ID: <4A37DA6D.1040703@mrabarnett.plus.com> Ron Adam wrote: > > > Ben Finney wrote: >> Carl Johnson >> writes: >> >>> we could introduce an empty set-literal and an odict-literal, and add >>> a more explicit form to replace the existing set literal. >> >> What do you mean by ?more explicit?? The existing set literal syntax >> is quite explicit. >> >>> s{} could be the empty set, o{} could be an empty odict, and we could >>> leave {} alone as the form for dicts. So, an odict literal would look >>> like o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), >>> ('b', '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could >>> (optionally?) have a little s{'a', 'c', 'b'} to make it more explicit >>> that this is a set, not a dict. >> >> I don't think that word ?explicit? means what you think it means. > > I think Carl is thinking that it is more concise and easier to read. > > When you read OrderedDict([('a', '1'),('b', '2'), ('c', '3')]), your > mind is parsing *Ordered Dictionary from list of these three tuples*. > > Whebn you read *o{'a':'1', 'b':'2', 'c':'3'}*, your mind does not have > the extra noise of converting the argument list of the constructor. > How about ['a':'1', 'b':'2', 'c':'3']? > > I think it really depends on how often ordered dictionaries literals > will be used weather or not it is worth doing. It also uses about 30% > to 50% less keystrokes. > Ideally we would have [...] for ordered and {...} for unordered, with ':' if it's keyed: [] ordered, value (list) [:] ordered, key+value (odict) {} unordered, value (set) {:} unordered, key+value (dict) It's too late to change that in Python 3.x, so it would have to be: [] ordered, value (list) [:] ordered, key+value (odict) {,} unordered, value (set) {} unordered, key+value (dict) From tjreedy at udel.edu Tue Jun 16 20:24:06 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jun 2009 14:24:06 -0400 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> Message-ID: Carl Johnson wrote: > I thought of this in the shower. It might not be a good idea, but I'd > like to hear what other people think. On analogy with b"", r"", etc. > we could introduce an empty set-literal and an odict-literal, and add > a more explicit form to replace the existing set literal. s{} could be > the empty set, o{} could be an empty odict, and we could leave {} > alone as the form for dicts. So, an odict literal would look like > o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), ('b', > '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could (optionally?) > have a little s{'a', 'c', 'b'} to make it more explicit that this is a > set, not a dict. > > I suppose if we wanted to get crazy, we could also have a frozenset > literal, fs{}? > > So what do people think? Is this too ugly to do? Does it confuse users > who are used to C-style braces? Or is it a logical extension of the > b"", r"", etc. system that could help make things follow EIBI better? I consider b"" and r"" to be hacks, necessary but not to be imitated without such necessity. I believe both affect how the lexer interprets code characters when it constructs literal value tokens to be fed to the parser. Collection-content displays are not, in general, constant and generally produce run-time code. From tjreedy at udel.edu Tue Jun 16 20:28:29 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jun 2009 14:28:29 -0400 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <4A37DA6D.1040703@mrabarnett.plus.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> <4A37DA6D.1040703@mrabarnett.plus.com> Message-ID: >> When you read OrderedDict([('a', '1'),('b', '2'), ('c', '3')]), OrderedDict({'a':1', 'b':'2', 'c':'3'}] > How about ['a':'1', 'b':'2', 'c':'3']? Nice Idea! tjr From tjreedy at udel.edu Tue Jun 16 20:25:22 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jun 2009 14:25:22 -0400 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <4A37DA6D.1040703@mrabarnett.plus.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> <4A37DA6D.1040703@mrabarnett.plus.com> Message-ID: MRAB wrote: > Ron Adam wrote: >> >> >> Ben Finney wrote: >>> Carl Johnson >>> writes: >>> >>>> we could introduce an empty set-literal and an odict-literal, and add >>>> a more explicit form to replace the existing set literal. >>> >>> What do you mean by ?more explicit?? The existing set literal syntax >>> is quite explicit. >>> >>>> s{} could be the empty set, o{} could be an empty odict, and we could >>>> leave {} alone as the form for dicts. So, an odict literal would look >>>> like o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), >>>> ('b', '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could >>>> (optionally?) have a little s{'a', 'c', 'b'} to make it more explicit >>>> that this is a set, not a dict. >>> >>> I don't think that word ?explicit? means what you think it means. >> >> I think Carl is thinking that it is more concise and easier to read. >> >> When you read OrderedDict([('a', '1'),('b', '2'), ('c', '3')]), your >> mind is parsing *Ordered Dictionary from list of these three tuples*. >> >> Whebn you read *o{'a':'1', 'b':'2', 'c':'3'}*, your mind does not have >> the extra noise of converting the argument list of the constructor. >> > How about ['a':'1', 'b':'2', 'c':'3']? > >> >> I think it really depends on how often ordered dictionaries literals >> will be used weather or not it is worth doing. It also uses about 30% >> to 50% less keystrokes. >> > Ideally we would have [...] for ordered and {...} for unordered, with > ':' if it's keyed: > > [] ordered, value (list) > [:] ordered, key+value (odict) > {} unordered, value (set) > {:} unordered, key+value (dict) > > It's too late to change that in Python 3.x, so it would have to be: > > [] ordered, value (list) > [:] ordered, key+value (odict) > {,} unordered, value (set) > {} unordered, key+value (dict) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From guido at python.org Tue Jun 16 20:36:44 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Jun 2009 11:36:44 -0700 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> Message-ID: On Tue, Jun 16, 2009 at 11:24 AM, Terry Reedy wrote: > Carl Johnson wrote: >> >> I thought of this in the shower. It might not be a good idea, but I'd >> like to hear what other people think. On analogy with b"", r"", etc. >> we could introduce an empty set-literal and an odict-literal, and add >> a more explicit form to replace the existing set literal. s{} could be >> the empty set, o{} could be an empty odict, and we could leave {} >> alone as the form for dicts. So, an odict literal would look like >> o{'a':'1', 'b':'2', 'c':'3'} instead of OrderedDict([('a', '1'), ('b', >> '2'), ('c', '3')]). And the set {'a', 'c', 'b'} could (optionally?) >> have a little s{'a', 'c', 'b'} to make it more explicit that this is a >> set, not a dict. >> >> I suppose if we wanted to get crazy, we could also have a frozenset >> literal, fs{}? >> >> So what do people think? Is this too ugly to do? Does it confuse users >> who are used to C-style braces? Or is it a logical extension of the >> b"", r"", etc. system that could help make things follow EIBI better? -1 from me. > I consider b"" and r"" to be hacks, necessary but not to be imitated without > such necessity. I believe both affect how the lexer interprets code > characters when it constructs literal value tokens to be fed to the parser. Not quite -- the lexer passes the entire string literal uninterpreted to the parser (including quotes). It doesn't even need the 'b' or 'r' prefix to tell where the literal ends (though it does need the opening quote for that, and whether it's tripled). The code generation stage, however, looks at the prefix and interprets the contents of the literal differently based on the presence of 'b', and creates a different object depending on the presence of 'b'. (Or 'u' in Python 2.) > Collection-content displays are not, in general, constant and generally > produce run-time code. (?) > OrderedDict({'a':1', 'b':'2', 'c':'3'}] > > >> How about ['a':'1', 'b':'2', 'c':'3']? -100. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From denis.spir at free.fr Tue Jun 16 20:49:38 2009 From: denis.spir at free.fr (spir) Date: Tue, 16 Jun 2009 20:49:38 +0200 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <4A37DA6D.1040703@mrabarnett.plus.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> <4A37DA6D.1040703@mrabarnett.plus.com> Message-ID: <20090616204938.2078cd7e@o> Le Tue, 16 Jun 2009 18:46:21 +0100, MRAB s'exprima ainsi: > How about ['a':'1', 'b':'2', 'c':'3']? (for ordered dict literal) I really find this sensible. We have then: * [a, b, c] & [k1:v1, k2:v2, k3:v3] for ordered containers * {a, b, c} & {k1:v1, k2:v2, k3:v3} for unordered ones +1.0 for me Update: confusion of 1-item ordered dict with slice? Or just the same as confusion of 1-item list with index? (Note: have always found that [] and [:] for index and slice was somewhat wrong). Denis ------ la vita e estrany From tjreedy at udel.edu Tue Jun 16 22:19:36 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jun 2009 16:19:36 -0400 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> Message-ID: Guido van Rossum wrote: > On Tue, Jun 16, 2009 at 11:24 AM, Terry Reedy wrote: >> Collection-content displays are not, in general, constant and generally >> produce run-time code. > (?) Somewhat expanded, I meant this: Literals represent constants. They are sensibly interpreted into constants sometime before runtime, even if at a later stage than I thought. Displays, in general, can contain variables and expressions that can only be evaluated at runtime. Therefore, the object they represent must, in general, be evaluated at runtime. (The pre-computation of tuples that only contain constants is an implementation-specific special case optimization.) Code literals and code displays are, in this way among others, different beasts. Therefore, a device used for literals is not necessarily a good device for displays. The latter use needs more justification than the precedent of the former use. The OP has not provided such justification for an admittedly ugly device. -1 (which we agree on) Terry Jan Reedy From algorias at gmail.com Tue Jun 16 23:25:40 2009 From: algorias at gmail.com (Vitor Bosshard) Date: Tue, 16 Jun 2009 17:25:40 -0400 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> <4A37DA6D.1040703@mrabarnett.plus.com> Message-ID: <2987c46d0906161425r1abb58edobc509c8041984de9@mail.gmail.com> 2009/6/16 Terry Reedy > > When you read OrderedDict([('a', '1'),('b', '2'), ('c', '3')]), >>> >> > OrderedDict({'a':1', 'b':'2', 'c':'3'}] > Wouldn't you lose the ordering in that case? OrderedDict({'c':'3', 'a':'1', 'b':'2'}) produces the same result as the above. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmjohnson.mailinglist at gmail.com Tue Jun 16 23:29:41 2009 From: cmjohnson.mailinglist at gmail.com (Carl Johnson) Date: Tue, 16 Jun 2009 11:29:41 -1000 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <4A37ABE2.6070302@ronadam.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> Message-ID: <3bdda690906161429t2d1bf82j3987c7c0f93d206b@mail.gmail.com> Ron Adam wrote: > Ben Finney wrote: >> Carl Johnson writes: >>> we could introduce an empty set-literal and an odict-literal, and add >>> a more explicit form to replace the existing set literal. >> >> What do you mean by ?more explicit?? The existing set literal syntax >> is quite explicit. > I think Carl is thinking that it is more concise and easier to read. Yes, that's part of it. Thank you. What I mean by "explicit" in this case is something like "harder to accidentally misparse in your mind." So for example if you see {f(i) for i in my_data} obviously it is unambiguously a set-comprehension, but if you aren't paying close enough attention you might mistakenly think it was a dict-comprehension. On the other hand, s{f(i) for i in my_data} more or less screams, "I am a set-comprehension!" On the other hand, it's ugly, and the BDFL doesn't like it. So, it's probably a dead idea now? Terry Reedy wrote: > Literals represent constants. They are sensibly interpreted into constants > sometime before runtime, even if at a later stage than I thought. I'm not sure what you're getting at here. What about d = {} ? That's not immutable. Nor is l = [1, 2, 3]. Or by "constant" do you just mean something that will be the same at the *start* of every program, but it may or may not change before the program is over? - - - - I think ["a": 1, "b": 2] is an interesting syntax, and certainly more elegant than o{"a": 1, "b": 2} but I worry that it could be ambiguous. I can't think of any way to do it off the top of my head, but might there be some valid way of writing l[1:2] that could be ambiguous between the two? If nothing else, it would confuse newbies about what [:] means: slice or odict? -- Carl From python at mrabarnett.plus.com Tue Jun 16 23:56:39 2009 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 16 Jun 2009 22:56:39 +0100 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <20090616204938.2078cd7e@o> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> <4A37DA6D.1040703@mrabarnett.plus.com> <20090616204938.2078cd7e@o> Message-ID: <4A381517.60205@mrabarnett.plus.com> spir wrote: > Le Tue, 16 Jun 2009 18:46:21 +0100, > MRAB s'exprima ainsi: > >> How about ['a':'1', 'b':'2', 'c':'3']? > (for ordered dict literal) > > I really find this sensible. > We have then: > * [a, b, c] & [k1:v1, k2:v2, k3:v3] for ordered containers > * {a, b, c} & {k1:v1, k2:v2, k3:v3} for unordered ones > > +1.0 for me > > Update: confusion of 1-item ordered dict with slice? Or just the same as confusion of 1-item list with index? (Note: have always found that [] and [:] for index and slice was somewhat wrong). > Can you confuse a 1-item list with an index? No, because the syntax/grammar disambiguates them. Is that also the case with a 1-item ordered dict vs a slice? In Python 2.6, [0 : 1] (== slice(0, 1)) raises SyntaxError. Does that also occur in Python 3.x? From python at mrabarnett.plus.com Wed Jun 17 00:00:05 2009 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 16 Jun 2009 23:00:05 +0100 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <3bdda690906161429t2d1bf82j3987c7c0f93d206b@mail.gmail.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> <3bdda690906161429t2d1bf82j3987c7c0f93d206b@mail.gmail.com> Message-ID: <4A3815E5.5040202@mrabarnett.plus.com> Carl Johnson wrote: > Ron Adam wrote: >> Ben Finney wrote: >>> Carl Johnson writes: >>>> we could introduce an empty set-literal and an odict-literal, and add >>>> a more explicit form to replace the existing set literal. >>> What do you mean by ?more explicit?? The existing set literal syntax >>> is quite explicit. >> I think Carl is thinking that it is more concise and easier to read. > > Yes, that's part of it. Thank you. What I mean by "explicit" in this > case is something like "harder to accidentally misparse in your mind." > So for example if you see {f(i) for i in my_data} obviously it is > unambiguously a set-comprehension, but if you aren't paying close > enough attention you might mistakenly think it was a > dict-comprehension. On the other hand, s{f(i) for i in my_data} more > or less screams, "I am a set-comprehension!" On the other hand, it's > ugly, and the BDFL doesn't like it. So, it's probably a dead idea now? > > Terry Reedy wrote: >> Literals represent constants. They are sensibly interpreted into constants >> sometime before runtime, even if at a later stage than I thought. > > I'm not sure what you're getting at here. What about d = {} ? That's > not immutable. Nor is l = [1, 2, 3]. Or by "constant" do you just mean > something that will be the same at the *start* of every program, but > it may or may not change before the program is over? > > - - - - > > I think ["a": 1, "b": 2] is an interesting syntax, and certainly more > elegant than o{"a": 1, "b": 2} but I worry that it could be ambiguous. > I can't think of any way to do it off the top of my head, but might > there be some valid way of writing l[1:2] that could be ambiguous > between the two? If nothing else, it would confuse newbies about what > [:] means: slice or odict? > What about l[2]? Is [2] an index or a list? From steve at pearwood.info Wed Jun 17 01:40:10 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Jun 2009 09:40:10 +1000 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <4A37DA6D.1040703@mrabarnett.plus.com> Message-ID: <200906170940.11284.steve@pearwood.info> On Wed, 17 Jun 2009 04:28:29 am Terry Reedy wrote: > >> When you read OrderedDict([('a', '1'),('b', '2'), ('c', '3')]), > > OrderedDict({'a':1', 'b':'2', 'c':'3'}] Creating an ordered dict from arbitrarily ordered input data surely ranks up there with one of the most pointless operations ever! You know, this is such a simple, yet silly, mistake to make, I'd be almost tempted to disallow constructing an OrderedDict from regular dicts. If you want non-arbitrary ordering, you can't use a dict as input, and if you don't want arbitrary ordering, there's no point in using an ordered dict! As far as I can see, there is no use-case for OrderedDict(dict) other than to generate confused newbies. > > How about ['a':'1', 'b':'2', 'c':'3']? > > Nice Idea! I hope you're being sarcastic. -- Steven D'Aprano From python at mrabarnett.plus.com Wed Jun 17 02:13:27 2009 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 17 Jun 2009 01:13:27 +0100 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <200906170940.11284.steve@pearwood.info> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <4A37DA6D.1040703@mrabarnett.plus.com> <200906170940.11284.steve@pearwood.info> Message-ID: <4A383527.9030906@mrabarnett.plus.com> Steven D'Aprano wrote: > On Wed, 17 Jun 2009 04:28:29 am Terry Reedy wrote: >>>> When you read OrderedDict([('a', '1'),('b', '2'), ('c', '3')]), >> OrderedDict({'a':1', 'b':'2', 'c':'3'}] > > Creating an ordered dict from arbitrarily ordered input data surely > ranks up there with one of the most pointless operations ever! > > You know, this is such a simple, yet silly, mistake to make, I'd be > almost tempted to disallow constructing an OrderedDict from regular > dicts. If you want non-arbitrary ordering, you can't use a dict as > input, and if you don't want arbitrary ordering, there's no point in > using an ordered dict! > > As far as I can see, there is no use-case for OrderedDict(dict) other > than to generate confused newbies. > > > >>> How about ['a':'1', 'b':'2', 'c':'3']? >> Nice Idea! > > I hope you're being sarcastic. > If hard-coded ordered dicts are common enough then having syntactic sugar for them would be a good idea: ['a':'1', 'b':'2', 'c':'3'] is both shorter and clearer than OrderedDict([('a', '1'),('b', '2'), ('c', '3')]). I doesn't feel that unPythonic to me. But ordered dicts tend to be built one item at a time while iterating through some data whose values aren't known until runtime, so I doubt that ordered dict literals would be common enough anyway to warrant addition. From ben+python at benfinney.id.au Wed Jun 17 02:22:33 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 17 Jun 2009 10:22:33 +1000 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87prd42sgz.fsf@benfinney.id.au> <4A37ABE2.6070302@ronadam.com> <4A37DA6D.1040703@mrabarnett.plus.com> Message-ID: <87ljnr3e0m.fsf@benfinney.id.au> MRAB writes: > [for a literal OrderedDict,] How about ['a':'1', 'b':'2', 'c':'3']? +1. That's quite good, and has symmetry with existing syntax:: {'fred', 'barney', 4.75} # unordered set ['fred', 'barney', 4.75] # ordered list {'foo': 10.2, 'bar': 8.19} # unordered dict ['foo': 10.2, 'bar': 8.19] # ordered dict That looks good to me. > Ideally we would have [...] for ordered and {...} for unordered, with > ':' if it's keyed: > > [] ordered, value (list) > [:] ordered, key+value (odict) > {} unordered, value (set) > {:} unordered, key+value (dict) > > It's too late to change that in Python 3.x, so it would have to be: > > [] ordered, value (list) > [:] ordered, key+value (odict) > {,} unordered, value (set) > {} unordered, key+value (dict) -0. Unlike syntax for creating and populating an instance, I actually don't think we need to worry about having specific syntax for empty. Compared to a non-empty container, it's quite a lower burden to simply have an explicit call to the type constructor to create an empty container. -- \ ?Probably the toughest time in anyone's life is when you have | `\ to murder a loved one because they're the devil.? ?Emo Philips | _o__) | Ben Finney From benjamin at python.org Wed Jun 17 02:22:54 2009 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 17 Jun 2009 00:22:54 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Possible_method_of_distinguishing_betwee?= =?utf-8?q?n=09set-literals=2C_dict-literals=2C_and_odict-literals?= References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> Message-ID: Guido van Rossum writes: > On Tue, Jun 16, 2009 at 11:24 AM, Terry Reedy wrote: > > I consider b"" and r"" to be hacks, necessary but not to be imitated without > > such necessity. I believe both affect how the lexer interprets code > > characters when it constructs literal value tokens to be fed to the parser. > > Not quite -- the lexer passes the entire string literal uninterpreted > to the parser (including quotes). It doesn't even need the 'b' or 'r' > prefix to tell where the literal ends (though it does need the opening > quote for that, and whether it's tripled). The code generation stage, > however, looks at the prefix and interprets the contents of the > literal differently based on the presence of 'b', and creates a > different object depending on the presence of 'b'. (Or 'u' in Python > 2.) Technically, string decoding is done during the building of the AST not code generation. See parsestrplus in ast.c if you want to know all the gory details. Regards, Benjamin From greg.ewing at canterbury.ac.nz Wed Jun 17 02:30:47 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 17 Jun 2009 12:30:47 +1200 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <200906170940.11284.steve@pearwood.info> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <4A37DA6D.1040703@mrabarnett.plus.com> <200906170940.11284.steve@pearwood.info> Message-ID: <4A383937.5030002@canterbury.ac.nz> Steven D'Aprano wrote: > As far as I can see, there is no use-case for OrderedDict(dict) other > than to generate confused newbies. It's possible that you don't care about the order of the initial items, but want to preserve the ordering of items added later. Not sure how *likely* a requirement this is, though... -- Greg From ben+python at benfinney.id.au Wed Jun 17 02:25:33 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 17 Jun 2009 10:25:33 +1000 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> Message-ID: <87hbyf3dvm.fsf@benfinney.id.au> Guido van Rossum writes: > On Tue, Jun 16, 2009 at 11:24 AM, Terry Reedy wrote: > > OrderedDict({'a':1', 'b':'2', 'c':'3'}] > > > > > >> How about ['a':'1', 'b':'2', 'c':'3']? > > -100. (Hey! I though the valid range of votes was -1 through +1, I didn't know we were giving the BDFL more than one vote! :-) Can you summarise what you dislike about the above syntax suggestion for ordered dict literal? -- \ ?If the desire to kill and the opportunity to kill came always | `\ together, who would escape hanging?? ?Mark Twain, _Following | _o__) the Equator_ | Ben Finney From bruce at leapyear.org Wed Jun 17 02:38:16 2009 From: bruce at leapyear.org (Bruce Leban) Date: Tue, 16 Jun 2009 17:38:16 -0700 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <87hbyf3dvm.fsf@benfinney.id.au> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> Message-ID: Personally, if I had a time machine I would pick a syntax like set{...} over set([...]) but I don't have a time machine and even if I did I don't think I'd use it for that. The difference is that the former is special syntax, while the latter isn't. No matter what I do to the the varible r, r"string" is still a string, while set([...]) can be perverted. However, given what we've got I can't really say that ([ instead of { is that big a deal. --- Bruce On Tue, Jun 16, 2009 at 5:25 PM, Ben Finney > wrote: > Guido van Rossum writes: > > > On Tue, Jun 16, 2009 at 11:24 AM, Terry Reedy wrote: > > > OrderedDict({'a':1', 'b':'2', 'c':'3'}] > > > > > > > > >> How about ['a':'1', 'b':'2', 'c':'3']? > > > > -100. > > (Hey! I though the valid range of votes was -1 through +1, I didn't know > we were giving the BDFL more than one vote! :-) > > Can you summarise what you dislike about the above syntax suggestion for > ordered dict literal? > > -- > \ ?If the desire to kill and the opportunity to kill came always | > `\ together, who would escape hanging?? ?Mark Twain, _Following | > _o__) the Equator_ | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aahz at pythoncraft.com Wed Jun 17 03:12:10 2009 From: aahz at pythoncraft.com (Aahz) Date: Tue, 16 Jun 2009 18:12:10 -0700 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <87hbyf3dvm.fsf@benfinney.id.au> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> Message-ID: <20090617011210.GA9427@panix.com> On Wed, Jun 17, 2009, Ben Finney wrote: > Guido van Rossum writes: >> On Tue, Jun 16, 2009 at 11:24 AM, Terry Reedy wrote: >>> >>> OrderedDict({'a':1', 'b':'2', 'c':'3'}] >>> >>> >>>> How about ['a':'1', 'b':'2', 'c':'3']? >> >> -100. > > (Hey! I though the valid range of votes was -1 through +1, I didn't know > we were giving the BDFL more than one vote! :-) Guido can vote; he can also Pronounce. -100 is a shorthand idiom for "veto". (I know you were joking, but I figure there are probably people reading this who are not familiar with the Python development process or Guido's role in it.) -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Many customs in this life persist because they ease friction and promote productivity as a result of universal agreement, and whether they are precisely the optimal choices is much less important." --Henry Spencer From python at mrabarnett.plus.com Wed Jun 17 03:28:03 2009 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 17 Jun 2009 02:28:03 +0100 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> Message-ID: <4A3846A3.3090801@mrabarnett.plus.com> Bruce Leban wrote: > Personally, if I had a time machine I would pick a syntax like > set{...} > over > set([...]) > but I don't have a time machine and even if I did I don't think I'd use > it for that. The difference is that the former is special syntax, while > the latter isn't. No matter what I do to the the varible r, r"string" is > still a string, while set([...]) can be perverted. > > However, given what we've got I can't really say that ([ instead of { is > that big a deal. > [snip] You can have any iterable between the (...); it's just that the iterable happens to be a list literal. From stephen at xemacs.org Wed Jun 17 04:14:32 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Jun 2009 11:14:32 +0900 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <20090617011210.GA9427@panix.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> <20090617011210.GA9427@panix.com> Message-ID: <87ocsnk3nb.fsf@uwakimon.sk.tsukuba.ac.jp> Aahz writes: > On Wed, Jun 17, 2009, Ben Finney wrote: > > Guido van Rossum writes: > >> -100. > > > > (Hey! I though the valid range of votes was -1 through +1, I didn't know > > we were giving the BDFL more than one vote! :-) > > Guido can vote; he can also Pronounce. -100 is a shorthand idiom for > "veto". Erm, shorthand? But len("-100") == len("veto")? I actually read that differently (though I don't pretend to be capable of channeling anybody). When I see a "-100" from Guido (and to nearly the same extent, from any 'bot---do bots express emotion this way?--- or the FLUFL), I make a note that "this kind of proposal is not merely unacceptable, it's in bad taste." Then I try to figure out why.... From greg.ewing at canterbury.ac.nz Thu Jun 18 02:58:48 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 18 Jun 2009 12:58:48 +1200 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <20090617011210.GA9427@panix.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> <20090617011210.GA9427@panix.com> Message-ID: <4A399148.1060504@canterbury.ac.nz> Aahz wrote: > Guido can vote; he can also Pronounce. -100 is a shorthand idiom for > "veto". > > (I know you were joking, but I figure there are probably people reading > this who are not familiar with the Python development process or Guido's > role in it.) To further elucidate for those people, the +/-n numbers are not really votes -- nobody is adding them up and making a decision based solely on the total. They're just an expression of opinion, so you sometimes see things like -100 or -1000 to mean "I *really* don't like this". -- Greg From gerald.britton at gmail.com Thu Jun 18 15:51:42 2009 From: gerald.britton at gmail.com (Gerald Britton) Date: Thu, 18 Jun 2009 09:51:42 -0400 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <4A399148.1060504@canterbury.ac.nz> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> <20090617011210.GA9427@panix.com> <4A399148.1060504@canterbury.ac.nz> Message-ID: <5d1a32000906180651m79cc2ed1wc8773b1299765fd9@mail.gmail.com> ...and since the -100 is from Guido, you can basically forget it. It won't happen. On Wed, Jun 17, 2009 at 8:58 PM, Greg Ewing wrote: > Aahz wrote: > >> Guido can vote; he can also Pronounce. ?-100 is a shorthand idiom for >> "veto". >> >> (I know you were joking, but I figure there are probably people reading >> this who are not familiar with the Python development process or Guido's >> role in it.) > > To further elucidate for those people, the +/-n numbers > are not really votes -- nobody is adding them up and > making a decision based solely on the total. They're > just an expression of opinion, so you sometimes see > things like -100 or -1000 to mean "I *really* don't > like this". > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Gerald Britton From higerinbeijing at gmail.com Thu Jun 18 16:04:34 2009 From: higerinbeijing at gmail.com (zhong nanhai) Date: Thu, 18 Jun 2009 22:04:34 +0800 Subject: [Python-ideas] enhance filecmp to support text-and-universal-newline-mode file comparison Message-ID: Hello everyone: We know that we often use programmes or scripts to generate useful output files ,but these files may come from different OS platforms. Comparing them by reading line by line may seem a bit trivial and we can use the filecmp model to do such thing(the cmp function). But when we try to compare two text files from different platforms,e.g. Ubuntu and Windows, even though these two files contain the same content, the filecmp.cmp will return false.We know that the reason is different ways to handle newline flag in different platforms, '\n\r' for Windows,'\n' for Unix,'\r' for Mac, e.t.c. So is it a good idea to enhance the filecmp to support universal-newline-mode?If so, we can compare different files from different operation systems and if they have the same content, the filecmp.cmp would return true. Hoping everyone can give some advice about this idea. Thanks in advance, higer From python at mrabarnett.plus.com Thu Jun 18 16:08:32 2009 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 18 Jun 2009 15:08:32 +0100 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <5d1a32000906180651m79cc2ed1wc8773b1299765fd9@mail.gmail.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> <20090617011210.GA9427@panix.com> <4A399148.1060504@canterbury.ac.nz> <5d1a32000906180651m79cc2ed1wc8773b1299765fd9@mail.gmail.com> Message-ID: <4A3A4A60.4070703@mrabarnett.plus.com> Gerald Britton wrote: > ...and since the -100 is from Guido, you can basically forget it. It > won't happen. > Unless he gets hit by a bus, or a drive-by shooting by Larry Wall. :-) > On Wed, Jun 17, 2009 at 8:58 PM, Greg Ewing wrote: >> Aahz wrote: >> >>> Guido can vote; he can also Pronounce. -100 is a shorthand idiom for >>> "veto". >>> >>> (I know you were joking, but I figure there are probably people reading >>> this who are not familiar with the Python development process or Guido's >>> role in it.) >> To further elucidate for those people, the +/-n numbers >> are not really votes -- nobody is adding them up and >> making a decision based solely on the total. They're >> just an expression of opinion, so you sometimes see >> things like -100 or -1000 to mean "I *really* don't >> like this". >> >> -- >> Greg >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > > > From solipsis at pitrou.net Thu Jun 18 20:17:13 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 18 Jun 2009 18:17:13 +0000 (UTC) Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> <20090617011210.GA9427@panix.com> <4A399148.1060504@canterbury.ac.nz> <5d1a32000906180651m79cc2ed1wc8773b1299765fd9@mail.gmail.com> <4A3A4A60.4070703@mrabarnett.plus.com> Message-ID: MRAB writes: > > Gerald Britton wrote: > > ...and since the -100 is from Guido, you can basically forget it. It > > won't happen. > > > Unless he gets hit by a bus, or a drive-by shooting by Larry Wall. If people had to use Perl 6 when shooting each other, criminality problems would disappear instantly. Regards Antoine. From lie.1296 at gmail.com Thu Jun 18 22:19:01 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Fri, 19 Jun 2009 06:19:01 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional Message-ID: In list/generator comprehension, currently we have no way to access the result of the expression and have to write something like this: [f(x) for x in l if f(x) > 0] if f() is heavy or non-pure (i.e. have side effects), calling f() twice might be undesirable. Even if f() is not a function such as [x + 1 for x in l if x + 1 > 0] it looks ugly since we're repeating ourself. We can work around it like this: [y for y in (f(x) for x in l) if y > 0] but then we have an unnecessary nested loop comprehension. I'm suggesting about something like: [f(x) for x in l if @ > 0] where @ is the result of the listcomp's expression (i.e. f(x)) Personally, I don't like the use of symbols like @, as python is not perl. I'm still thinking of a better syntax/approach and is open for suggestion. What do you guys think? From aahz at pythoncraft.com Thu Jun 18 22:22:21 2009 From: aahz at pythoncraft.com (Aahz) Date: Thu, 18 Jun 2009 13:22:21 -0700 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: Message-ID: <20090618202221.GA19967@panix.com> On Fri, Jun 19, 2009, Lie Ryan wrote: > > In list/generator comprehension, currently we have no way to access the > result of the expression and have to write something like this: > > [f(x) for x in l if f(x) > 0] > > if f() is heavy or non-pure (i.e. have side effects), calling f() twice > might be undesirable. Listcomps and genexps are like lambdas: run up against their limits and you should switch to a regular for loop or generator. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From p.f.moore at gmail.com Thu Jun 18 22:37:38 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 18 Jun 2009 21:37:38 +0100 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: Message-ID: <79990c6b0906181337m3d340b7dq51288c717fbb40b0@mail.gmail.com> 2009/6/18 Lie Ryan : > In list/generator comprehension, currently we have no way to access the > result of the expression and have to write something like this: > > [f(x) for x in l if f(x) > 0] > > if f() is heavy or non-pure (i.e. have side effects), calling f() twice > might be undesirable. [y for y in (f(x) for x in l) if y > 0] But as Aahz says, if it's too complex, use an explicit loop. Not everything needs to be a one-liner. Paul From python at rcn.com Thu Jun 18 22:43:09 2009 From: python at rcn.com (Raymond Hettinger) Date: Thu, 18 Jun 2009 13:43:09 -0700 Subject: [Python-ideas] Accessing the result of comprehension's expressionfrom the conditional References: Message-ID: [Lie Ryan] > In list/generator comprehension, currently we have no way to access the > result of the expression and have to write something like this: > > [f(x) for x in l if f(x) > 0] > > if f() is heavy or non-pure (i.e. have side effects), calling f() twice > might be undesirable. [x for x in map(f, l) if x > 0] Raymond From tjreedy at udel.edu Thu Jun 18 23:02:12 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 18 Jun 2009 17:02:12 -0400 Subject: [Python-ideas] Aid reiteration with new class: gfic Message-ID: Backgound: Most functions that take an iterable as parameter iterate just once. Users of the function can pass any iterable, including iterators. The function will call iter(imput) and go. If the user wants to iterate thru a virtual collection defined by an instance of an iterator class (built-in or user-defined) or generator function, the user must call the gf/ic with the appropriate arguments and pass the result. Problem: Some functions iterate thru the input more than once. If the input is not an iterator, there is no problem: call iter(input) again. If the input is an iterator, that just returns the now-empty iterator. Bad. General solution: The most general solution is for the caller to pass the result of list(it) instead of it. If the caller starts with only an iterator, that is the only solution I know of. Special solution: If the caller starts with an non-iterable iterator source -- iterator class or generator function (which can be regarded as defining a virtual subclass of iterator class 'generator') -- it might be better to package that source and required args as an iterable. class gfic: #untested as yet def __init__(self, func, *args, **kwds): self.func = func self.args = args self.kwds = kwds self.__name__ = func.__name__ def __iter__(self): return self.func(*self.args, **self.kwds) This is similar, I believe, to functools.partial except that the wrapping is total rather than partial and the wrapped call is in __iter__ instead of __call__. Would this be a sensible addition to functools, say, or is the use case too rare? Terry Jan Reedy From g.brandl at gmx.net Fri Jun 19 00:19:31 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 19 Jun 2009 00:19:31 +0200 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: Message-ID: Terry Reedy schrieb: > Would this be a sensible addition to functools, say, or is the use case > too rare? I suggest you find a better name before inclusion though :) If you can find none, it might not be a sensible addition... Georg From steder at gmail.com Fri Jun 19 02:59:59 2009 From: steder at gmail.com (Mike Steder) Date: Thu, 18 Jun 2009 19:59:59 -0500 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: Message-ID: <147D79BA-D96D-464A-A472-8C589EC99A0B@gmail.com> On Jun 18, 2009, at 4:02 PM, Terry Reedy wrote: > > This is similar, I believe, to functools.partial except that the > wrapping is total rather than partial and the wrapped call is in > __iter__ instead of __call__. > > Would this be a sensible addition to functools, say, or is the use > case too rare? > My first thought on reading this was that it reminded me of itertools.cycle so it might be a reasonable fit in itertools. I'd call it a "reiterable" or perhaps "reiterator". Your untested sample code worked nicely for me. ;-) ~Mike From steve at pearwood.info Fri Jun 19 05:11:11 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 19 Jun 2009 13:11:11 +1000 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: Message-ID: <200906191311.12155.steve@pearwood.info> On Fri, 19 Jun 2009 07:02:12 am Terry Reedy wrote: > Backgound: > Most functions that take an iterable as parameter iterate just once. > Users of the function can pass any iterable, including iterators. > The function will call iter(imput) and go. > > If the user wants to iterate thru a virtual collection defined by an > instance of an iterator class (built-in or user-defined) or generator > function, the user must call the gf/ic with the appropriate arguments > and pass the result. I don't think it's an undue burden on the caller to construct their own iterator before calling your function. To summarise your idea (correct me if I'm wrong): There are three ways of passing iterable-like arguments: (A) pass a sequence object like lists; (B) pass generators or iterators; (C) pass a constructor which returns a generator or iterator, plus appropriate arguments. Insider your function, you can easily iterate over (A) or (C) multiple times, but not (B). I don't think there's any way to treat all three cases identically. Your proposed gfic class will allow you to treat case (C) just like (A), at the cost of expecting the caller to call gfic() rather than pass a constructor directly. But the caller still can't pass an iterator in place of a sequence or constructor, so you haven't solved anything, merely shifted the burden on the user from calling one of: function(list(constructor(*args, **kwargs))) function(list(iterator)) to calling one of: function(gfic(constructor, *args, **kwargs)) function(list(iterator)) As I see it, the correct solution for "my function needs to iterate over an iterable twice" is not to expect the caller to pass a sequence, but to convert the iterable to a sequence inside your function: def function(iterable): # Iterate over iterable twice L = list(iterable) for _ in (1, 2): for x in L: pass Apart from needing to knowing to avoid non-terminating iterators, the user need not know whether you walk the input once or twice. I suppose there is some benefit when dealing with huge iterators, but that's probably best dealt with on an ad hoc basis by requiring the user to pass a constructor directly. -- Steven D'Aprano From tjreedy at udel.edu Fri Jun 19 08:38:30 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Jun 2009 02:38:30 -0400 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <200906191311.12155.steve@pearwood.info> References: <200906191311.12155.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > On Fri, 19 Jun 2009 07:02:12 am Terry Reedy wrote: >> Backgound: >> Most functions that take an iterable as parameter iterate just once. >> Users of the function can pass any iterable, including iterators. >> The function will call iter(imput) and go. >> >> If the user wants to iterate thru a virtual collection defined by an >> instance of an iterator class (built-in or user-defined) or generator >> function, the user must call the gf/ic with the appropriate arguments >> and pass the result. > > I don't think it's an undue burden on the caller to construct their own > iterator before calling your function. Only if an iterator is not allowed. Nor do I think it an undue burden either to wrap the constructor, when appropriate, instead of calling it. > To summarise your idea (correct me if I'm wrong): > > There are three ways of passing iterable-like arguments: > > (A) pass a sequence object like lists; > > (B) pass generators or iterators; > > (C) pass a constructor which returns a generator or iterator, plus > appropriate arguments. Right. > > Insider your function, you can easily iterate over (A) or (C) multiple > times, but not (B). > > > I don't think there's any way to treat all three cases identically. Your > proposed gfic class will allow you to treat case (C) just like (A), That is precisely the point. > at > the cost of expecting the caller to call gfic() rather than pass a > constructor directly. See above > But the caller still can't pass an iterator in > place of a sequence or constructor, so you haven't solved anything, > merely shifted the burden on the user from calling one of: > > function(list(constructor(*args, **kwargs))) > function(list(iterator)) > > to calling one of: > > function(gfic(constructor, *args, **kwargs)) > function(list(iterator)) Right. I consider list-avoidance a good thing, more so, it appears than you ;-). > > As I see it, the correct solution for "my function needs to iterate over > an iterable twice" is not to expect the caller to pass a sequence, but > to convert the iterable to a sequence inside your function: > > def function(iterable): > # Iterate over iterable twice > L = list(iterable) > for _ in (1, 2): > for x in L: > pass > > Apart from needing to knowing to avoid non-terminating iterators, the > user need not know whether you walk the input once or twice. > > I suppose there is some benefit when dealing with huge iterators, but > that's probably best dealt with on an ad hoc basis by requiring the > user to pass a constructor directly. Thank you for your response and analysis. tjr From tjreedy at udel.edu Fri Jun 19 08:41:01 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Jun 2009 02:41:01 -0400 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <147D79BA-D96D-464A-A472-8C589EC99A0B@gmail.com> References: <147D79BA-D96D-464A-A472-8C589EC99A0B@gmail.com> Message-ID: Mike Steder wrote: > > On Jun 18, 2009, at 4:02 PM, Terry Reedy wrote: >> >> This is similar, I believe, to functools.partial except that the >> wrapping is total rather than partial and the wrapped call is in >> __iter__ instead of __call__. >> >> Would this be a sensible addition to functools, say, or is the use >> case too rare? >> > > My first thought on reading this was that it reminded me of > itertools.cycle so it might be a reasonable fit in itertools. > > I'd call it a "reiterable" or perhaps "reiterator". I like that better. > > Your untested sample code worked nicely for me. ;-) Great. From lie.1296 at gmail.com Fri Jun 19 09:01:13 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Fri, 19 Jun 2009 17:01:13 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <79990c6b0906181337m3d340b7dq51288c717fbb40b0@mail.gmail.com> References: <79990c6b0906181337m3d340b7dq51288c717fbb40b0@mail.gmail.com> Message-ID: Paul Moore wrote: > 2009/6/18 Lie Ryan : >> In list/generator comprehension, currently we have no way to access the >> result of the expression and have to write something like this: >> >> [f(x) for x in l if f(x) > 0] >> >> if f() is heavy or non-pure (i.e. have side effects), calling f() twice >> might be undesirable. > > [y for y in (f(x) for x in l) if y > 0] > > But as Aahz says, if it's too complex, use an explicit loop. Not > everything needs to be a one-liner. > > Paul Avoiding a nested list comprehension is exactly the motivation for the idea. From lie.1296 at gmail.com Fri Jun 19 09:39:32 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Fri, 19 Jun 2009 17:39:32 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <20090618202221.GA19967@panix.com> References: <20090618202221.GA19967@panix.com> Message-ID: Aahz wrote: > On Fri, Jun 19, 2009, Lie Ryan wrote: >> In list/generator comprehension, currently we have no way to access the >> result of the expression and have to write something like this: >> >> [f(x) for x in l if f(x) > 0] >> >> if f() is heavy or non-pure (i.e. have side effects), calling f() twice >> might be undesirable. > > Listcomps and genexps are like lambdas: run up against their limits and > you should switch to a regular for loop or generator. I think of it not as limitation but as an odd gap in functionality. I think having the semantics that the filtering is done after the expression part would be much more useful than the current behavior (filtering before expression). If filtering is done after expression, we can access both the original and the transformed objects (it may also be possible to optimize the cases where the filter does not use the transformed objects, although this complex behavior wouldn't be pythonic) Try rewriting this: res = [x**x as F for x in nums if F < 100] (note: this is my new preferred syntax) Attempts: - Using a regular for-loop res = [] for x in nums: F = x**x if F < 100: res.append(F) remarks: five lines that's much more difficult to understand than a single, concise expression in standard form. - Using nested comprehension res = [F for F in (x**x for x in nums) if F < 100] remarks: using nested list comprehension is DRY (on a loose definition of DRY principle). I have to repeat the comprehension body twice, when I only need one expression and one filtering. - Using map() res = [F for F in map(lambda x: x**x, nums) if F < 100] remarks: when the expression part is just a simple expression, you have to use lambda and that's plain ugly. Not to mention when you want to filter based on both x and F. Advantages of the proposal: - shorter - faster, as looping is done in C - more readable. The main advantage of comprehension is that it have standardized form, which is easier to understand, unlike a for-loop which can have an infinite number of variations. - (unnecessary) nested comprehension is an abuse. - with `as` keyword, no new keyword and no ambiguity since currently `as` cannot exist inside comprehension. Disadvantages: - reverses the current semantic of filtering-then-expression. This shouldn't be too much problem since side-effect on the expression part is a cardinal sin and... - if the expression part is heavy, it might be possible to do optimization by filtering first when the filter part does not require the result (i.e. when there is no "as" clause). A good side effect of this optimization is codes that relies on filtering being done before expression will just work as they cannot contain an `as` keyword. (As "simple is better than complex", I actually don't really like `as` can change evaluation order; I much prefer to keep everything simple and consistent, i.e. always evaluate expression then filter or otherwise) possible syntaxes: - [x**x as F for x in nums if F < 100] the as keyword is already often used to rename things (in with, import, etc) I like this one much better than @. The as part, of course, is optional - [x**x for x in nums if @ < 100] the initial proposed syntax, ugly as hell. From pyideas at rebertia.com Fri Jun 19 10:04:18 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Fri, 19 Jun 2009 01:04:18 -0700 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <20090618202221.GA19967@panix.com> Message-ID: <50697b2c0906190104l20707359m27206d7d347de3d5@mail.gmail.com> On Fri, Jun 19, 2009 at 12:39 AM, Lie Ryan wrote: > Aahz wrote: >> On Fri, Jun 19, 2009, Lie Ryan wrote: >>> In list/generator comprehension, currently we have no way to access the >>> result of the expression and have to write something like this: >>> >>> [f(x) for x in l if f(x) > 0] >>> >>> if f() is heavy or non-pure (i.e. have side effects), calling f() twice >>> might be undesirable. >> >> Listcomps and genexps are like lambdas: run up against their limits and >> you should switch to a regular for loop or generator. > > I think of it not as limitation but as an odd gap in functionality. I > think having the semantics that the filtering is done after the > expression part would be much more useful than the current behavior > (filtering before expression). > > If filtering is done after expression, we can access both the original > and the transformed objects (it may also be possible to optimize the > cases where the filter does not use the transformed objects, although > this complex behavior wouldn't be pythonic) > > Try rewriting this: > > res = [x**x as F for x in nums if F < 100] > (note: this is my new preferred syntax) > > Attempts: > > - Using a regular for-loop > > res = [] > for x in nums: > F = x**x > if F < 100: > res.append(F) > > remarks: five lines that's much more difficult to understand than a > single, concise expression in standard form. Depends on how complicated the "standard form" is; right now, it (list comps) is/are relatively simple. Perl syntax is concise and adheres to a standard, but is not easy to understand. Not to equate a minor syntax addition to Perl, but every addition is a step in that general direction. The road to hell was paved with good intentions. Cheers, Chris -- http://blog.rebertia.com From p.f.moore at gmail.com Fri Jun 19 10:15:53 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 19 Jun 2009 09:15:53 +0100 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <20090618202221.GA19967@panix.com> Message-ID: <79990c6b0906190115q20a52a85y729f51173175b045@mail.gmail.com> 2009/6/19 Lie Ryan : > res = [x**x as F for x in nums if F < 100] > (note: this is my new preferred syntax) [...] > Advantages of the proposal: > - shorter > - faster, as looping is done in C > - more readable. The main advantage of comprehension is that it have > standardized form, which is easier to understand, unlike a for-loop > which can have an infinite number of variations. > - (unnecessary) nested comprehension is an abuse. > - with `as` keyword, no new keyword and no ambiguity since currently > `as` cannot exist inside comprehension. > > Disadvantages: > - reverses the current semantic of filtering-then-expression. This > shouldn't be too much problem since side-effect on the expression part > is a cardinal sin and... > - if the expression part is heavy, it might be possible to do > optimization by filtering first when the filter part does not require > the result (i.e. when there is no "as" clause). A good side effect of > this optimization is codes that relies on filtering being done before > expression will just work as they cannot contain an `as` keyword. > (As "simple is better than complex", I actually don't really like `as` > can change evaluation order; I much prefer to keep everything simple and > consistent, i.e. always evaluate expression then filter or otherwise) > > possible syntaxes: > - [x**x as F for x in nums if F < 100] > ?the as keyword is already often used to rename things > ?(in with, import, etc) I like this one much better than @. The as > ?part, of course, is optional > - [x**x for x in nums if @ < 100] > ?the initial proposed syntax, ugly as hell. OK, with this explanation (and the new syntax) I see what you're getting at better. However, changing the order of evaluate vs filter is a huge compatibility problem. There's no way this will be possible. Even with syntax triggering the change (so that it's one way with the "as", the other without), that's a disaster waiting to happen. You have at least 3 explicit ways of stating your intent (genexp inside listcomp, map inside listcomp, explicit loop). None is as clean-looking as your (amended) proposal, but they work now, and they don't have the semantic issues of your proposal. (For a more general, more radical, equally certain to be shot down, option, which at least doesn't introduce the change in semantics, you could try proposing "as" as an assignment-as-expression operator. So you could have [y for x in l if (f(x) as y) < 100] Hmm, on second thoughts - no, don't bother... :-)) Paul. From taleinat at gmail.com Fri Jun 19 13:34:03 2009 From: taleinat at gmail.com (Tal Einat) Date: Fri, 19 Jun 2009 14:34:03 +0300 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: Message-ID: <7afdee2f0906190434r5bcba151t64c80695fde7aa45@mail.gmail.com> Lie Ryan wrote: > In list/generator comprehension, currently we have no way to access the > result of the expression and have to write something like this: > > [f(x) for x in l if f(x) > 0] > > if f() is heavy or non-pure (i.e. have side effects), calling f() twice > might be undesirable. > > Even if f() is not a function such as > > [x + 1 for x in l if x + 1 > 0] > > it looks ugly since we're repeating ourself. > > We can work around it like this: > > [y for y in (f(x) for x in l) if y > 0] > > but then we have an unnecessary nested loop comprehension. > > I'm suggesting about something like: > > [f(x) for x in l if @ > 0] > > where @ is the result of the listcomp's expression (i.e. f(x)) > > Personally, I don't like the use of symbols like @, as python is not > perl. I'm still thinking of a better syntax/approach and is open for > suggestion. > > What do you guys think? IMO this is premature optimization. (I know that's an annoying thing to say, but bear with me...) Just using [f(x) for x in nums if f(x) > 0] is the most readable and obvious option. I think the case where such a line of code actually needs to be optimized is rare, and on there rare occasions using a slightly less readable variant is reasonable (along with an insightful comment). In other words, from my experience all we would gain from the proposed new syntax is making such premature optimization easier, at the cost of less readable code and more complex syntax. - Tal From jimjjewett at gmail.com Fri Jun 19 15:38:46 2009 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 19 Jun 2009 09:38:46 -0400 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <20090618202221.GA19967@panix.com> Message-ID: On Fri, Jun 19, 2009 at 3:39 AM, Lie Ryan wrote: > res = [x**x as F for x in nums if F < 100] This, I have wanted. That doesn't resolve all the issues others raised, but (at least for me), you just moved it from "a little worse than the status quo" to "hmm... that would be nice if it could be done without too many side effect on the rest of the language." On the possibility that a more general problem sometimes spurs a more elegant solution, I'll point out that I have more often wanted access to the previous or following element, and that my desired filters on the results are often reliant on the results-so-far. > Disadvantages: > - reverses the current semantic of filtering-then-expression. For What Its Worth, I'm not sure how strong that argument should be. Is this just a bizarre corner case, or is there lots of code that relies on it? That ordering actually surprises me, because I expect python to evaluate left to right. On the other hand, now that we have conditional expressions, consistency with those is probably more important, so maybe the number of people surprised by the current situation will go down with time. -jJ From jimjjewett at gmail.com Fri Jun 19 15:45:08 2009 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 19 Jun 2009 09:45:08 -0400 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <7afdee2f0906190434r5bcba151t64c80695fde7aa45@mail.gmail.com> References: <7afdee2f0906190434r5bcba151t64c80695fde7aa45@mail.gmail.com> Message-ID: On Fri, Jun 19, 2009 at 7:34 AM, Tal Einat wrote: > Just using [f(x) for x in nums if f(x) > 0] is the most readable and > obvious option. I would often prefer to break it into two steps: temp = (f(x) for x in nums) results = [e for e in temp if e>0] Others will dislike the extra line and temp var, which is one reason it isn't among the several solutions previously suggested. Are these differences big enough (or the solutions obscure enough) that the variation is itself a cost of the current situation? -jJ From jason.orendorff at gmail.com Fri Jun 19 16:17:53 2009 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Fri, 19 Jun 2009 09:17:53 -0500 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <20090618202221.GA19967@panix.com> Message-ID: On Fri, Jun 19, 2009 at 2:39 AM, Lie Ryan wrote: > res = [x**x as F for x in nums if F < 100] In some languages, you can use "let" in a list comprehension: [f | x <- nums, let f = x**x, f < 100] which I guess might look something like this in python: [f for x in nums let f = x**x if f < 100] (JavaScript actually has a `let` keyword so I suspect we will eventually adopt something like that.) -j From yoavglazner at gmail.com Fri Jun 19 16:28:54 2009 From: yoavglazner at gmail.com (yoav glazner) Date: Fri, 19 Jun 2009 17:28:54 +0300 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <200906191311.12155.steve@pearwood.info> Message-ID: On Fri, Jun 19, 2009 at 6:11 AM, Steven D'Aprano wrote: > On Fri, 19 Jun 2009 07:02:12 am Terry Reedy wrote: > > As I see it, the correct solution for "my function needs to iterate over > an iterable twice" is not to expect the caller to pass a sequence, but > to convert the iterable to a sequence inside your function: > > def function(iterable): > # Iterate over iterable twice > L = list(iterable) > for _ in (1, 2): > for x in L: > pass > As I see it the correct solution is def function(iterable): # Iterate over iterable twice L = iterable.clone() #can be done? for x in L: pass for x in iterable : pass this way i avoid endless iterators... -------------- next part -------------- An HTML attachment was scrubbed... URL: From taleinat at gmail.com Fri Jun 19 16:34:15 2009 From: taleinat at gmail.com (Tal Einat) Date: Fri, 19 Jun 2009 17:34:15 +0300 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <7afdee2f0906190434r5bcba151t64c80695fde7aa45@mail.gmail.com> Message-ID: <7afdee2f0906190734i4edeffd1pc69ceb5cfd03d256@mail.gmail.com> On Fri, Jun 19, 2009 at 4:45 PM, Jim Jewett wrote: > On Fri, Jun 19, 2009 at 7:34 AM, Tal Einat wrote: >> Just using [f(x) for x in nums if f(x) > 0] is the most readable and >> obvious option. > > I would often prefer to break it into two steps: > > ? ?temp = (f(x) for x in nums) > ? ?results = [e for e in temp if e>0] I sometimes find myself doing that too. That's probably more readable than what I wrote :) > Others will dislike the extra line and temp var, which is one reason > it isn't among the several solutions previously suggested. I sometimes avoid the temporary variable by using the same one twice, e.g.: results = (f(x) for x in nums) results = [res for res in results if res > 0] In the above example the second line is just filtering the results, and I feel this idiom conveys the idea pretty well. - Tal From steve at pearwood.info Fri Jun 19 16:35:49 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Jun 2009 00:35:49 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <20090618202221.GA19967@panix.com> Message-ID: <200906200035.51009.steve@pearwood.info> On Fri, 19 Jun 2009 05:39:32 pm Lie Ryan wrote: > Aahz wrote: > > On Fri, Jun 19, 2009, Lie Ryan wrote: > >> In list/generator comprehension, currently we have no way to > >> access the result of the expression and have to write something > >> like this: > >> > >> [f(x) for x in l if f(x) > 0] > >> > >> if f() is heavy or non-pure (i.e. have side effects), calling f() > >> twice might be undesirable. > > > > Listcomps and genexps are like lambdas: run up against their limits > > and you should switch to a regular for loop or generator. > > I think of it not as limitation but as an odd gap in functionality. I > think having the semantics that the filtering is done after the > expression part would be much more useful than the current behavior > (filtering before expression). The point of the filtering is to avoid needlessly calculating a potentially expensive expression only to throw it away. If you want expression first, then filter, you can get that already in a one-liner: filter(lambda x: x > 0, [f(x) for x in seq]) Don't create new syntax when there are perfectly good functions that do the job already. -- Steven D'Aprano From denis.spir at free.fr Fri Jun 19 17:08:32 2009 From: denis.spir at free.fr (spir) Date: Fri, 19 Jun 2009 17:08:32 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <7afdee2f0906190434r5bcba151t64c80695fde7aa45@mail.gmail.com> Message-ID: <20090619170832.53e0dd0e@o> Le Fri, 19 Jun 2009 09:45:08 -0400, Jim Jewett s'exprima ainsi: > On Fri, Jun 19, 2009 at 7:34 AM, Tal Einat wrote: > > Just using [f(x) for x in nums if f(x) > 0] is the most readable and > > obvious option. > > I would often prefer to break it into two steps: > > temp = (f(x) for x in nums) > results = [e for e in temp if e>0] > > Others will dislike the extra line and temp var, which is one reason > it isn't among the several solutions previously suggested. Are these > differences big enough (or the solutions obscure enough) that the > variation is itself a cost of the current situation? Ditto for me. A list comp with both computation and filtering is for me two ideas, two steps, so I write two lines anyway. This also has the advantage to disambiguate the order issue (computation or filter first?) and there is no risk of double side-effect (which anyway I never use, but who knows...). Denis ------ la vita e estrany From tjreedy at udel.edu Fri Jun 19 17:27:52 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 19 Jun 2009 11:27:52 -0400 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <200906191311.12155.steve@pearwood.info> Message-ID: yoav glazner wrote: > > On Fri, Jun 19, 2009 at 6:11 AM, Steven D'Aprano > > wrote: > > On Fri, 19 Jun 2009 07:02:12 am Terry Reedy wrote: > > As I see it, the correct solution for "my function needs to iterate over > an iterable twice" is not to expect the caller to pass a sequence, but > to convert the iterable to a sequence inside your function: > > def function(iterable): > # Iterate over iterable twice > L = list(iterable) > for _ in (1, 2): > for x in L: > pass > > > As I see it the correct solution is > def function(iterable): > # Iterate over iterable twice > L = iterable.clone() #can be done? For non-iterator iterables, it is not necessary. For iterators in general, it is difficult at best. For iterators returned by a constructor, it should be much easier to re-call the constructor. From gagsl-py2 at yahoo.com.ar Fri Jun 19 18:20:20 2009 From: gagsl-py2 at yahoo.com.ar (Gabriel Genellina) Date: Fri, 19 Jun 2009 13:20:20 -0300 Subject: [Python-ideas] enhance filecmp to support text-and-universal-newline-mode file comparison References: Message-ID: En Thu, 18 Jun 2009 11:04:34 -0300, zhong nanhai escribi?: > We know that we often use programmes or scripts to generate useful > output files ,but these files may come from different OS platforms. > > Comparing them by reading line by line may seem a bit trivial and we > can use the filecmp model to do such thing(the cmp function). But > when we try to compare two text files from different platforms,e.g. > Ubuntu and Windows, even though these two files contain the same > content, the filecmp.cmp will return false.We know that the reason is > different ways to handle newline flag in different platforms, '\n\r' > for Windows,'\n' for Unix,'\r' for Mac, e.t.c. > > So is it a good idea to enhance the filecmp to support > universal-newline-mode?If so, we can compare different files from > different operation systems and if they have the same content, the > filecmp.cmp would return true. With aid from itertools.izip_longest, it's a one-line recipe: py> print repr(open("one.txt","rb").read()) 'hello\nworld!\nlast line\n' py> print repr(open("two.txt","rb").read()) 'hello\r\nworld!\r\nlast line\r\n' py> import filecmp py> filecmp.cmp("one.txt", "two.txt", False) False py> from itertools import izip_longest py> f1 = open("one.txt", "rU") py> f2 = open("two.txt", "rU") py> py> print all(line1==line2 for line1,line2 in izip_longest(f1,f2)) True Currently filecmp considers both files as binary, not text; if they differ in size they're considered different and the contents are not even read. If you want a generic text-mode file comparison, there are other factors to consider in addition to line endings: character encoding, BOM, character case, whitespace... All of those may be considered "irrelevant differences" by some people. A generic text file comparison should take all of them into account. -- Gabriel Genellina From grosser.meister.morti at gmx.net Fri Jun 19 19:54:12 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Fri, 19 Jun 2009 19:54:12 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: Message-ID: <4A3BD0C4.9090007@gmx.net> You could write: [x|y <- l, x <- [f(y)], x > 0] Oh, wait. Thats Haskell. And even in haskell you would write: [x|x <- map f l, x > 0] In Python you can write: [x for x in map(f,l) if x > 0] In Python 2.x you may want to write: from itertools import imap [x for x in imap(f,l) if x > 0] A more SQL like approach that would fit somewhat with pythons syntax would be (as you can see its exactly the same lengths as the above but needs a new name): [f(x) as y for x in l if y > 0] Because in SQL you can write (IIRC): select f(x) as y from l where y > 0; Maybe something like .Nets LINQ would be a nice idea to integrate in python? -panzi Lie Ryan wrote: > In list/generator comprehension, currently we have no way to access the > result of the expression and have to write something like this: > > [f(x) for x in l if f(x) > 0] > > if f() is heavy or non-pure (i.e. have side effects), calling f() twice > might be undesirable. > > Even if f() is not a function such as > > [x + 1 for x in l if x + 1 > 0] > > it looks ugly since we're repeating ourself. > > We can work around it like this: > > [y for y in (f(x) for x in l) if y > 0] > > but then we have an unnecessary nested loop comprehension. > > I'm suggesting about something like: > > [f(x) for x in l if @ > 0] > > where @ is the result of the listcomp's expression (i.e. f(x)) > > Personally, I don't like the use of symbols like @, as python is not > perl. I'm still thinking of a better syntax/approach and is open for > suggestion. > > What do you guys think? > From arnodel at googlemail.com Fri Jun 19 21:12:51 2009 From: arnodel at googlemail.com (Arnaud Delobelle) Date: Fri, 19 Jun 2009 20:12:51 +0100 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: Message-ID: <7CB48A8E-B250-46E9-BD2F-4956A8DC751E@googlemail.com> On 18 Jun 2009, at 21:19, Lie Ryan wrote: > In list/generator comprehension, currently we have no way to access > the > result of the expression and have to write something like this: > > [f(x) for x in l if f(x) > 0] > > if f() is heavy or non-pure (i.e. have side effects), calling f() > twice > might be undesirable. > > Even if f() is not a function such as > > [x + 1 for x in l if x + 1 > 0] > > it looks ugly since we're repeating ourself. > > We can work around it like this: > > [y for y in (f(x) for x in l) if y > 0] > > but then we have an unnecessary nested loop comprehension You can write: [y for x in l for y in [f(x)] if y > 0] Which is even worse ;) -- Arnaud From john.a.graham at gmail.com Fri Jun 19 21:23:55 2009 From: john.a.graham at gmail.com (John Graham) Date: Fri, 19 Jun 2009 14:23:55 -0500 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <200906191311.12155.steve@pearwood.info> Message-ID: <4A3BE5CB.6090505@gmail.com> Terry Reedy wrote: > yoav glazner wrote: >> >> On Fri, Jun 19, 2009 at 6:11 AM, Steven D'Aprano > > wrote: >> >> On Fri, 19 Jun 2009 07:02:12 am Terry Reedy wrote: >> >> As I see it, the correct solution for "my function needs to >> iterate over >> an iterable twice" is not to expect the caller to pass a >> sequence, but >> to convert the iterable to a sequence inside your function: >> >> def function(iterable): >> # Iterate over iterable twice >> L = list(iterable) >> for _ in (1, 2): >> for x in L: >> pass >> >> >> As I see it the correct solution is >> def function(iterable): >> # Iterate over iterable twice >> L = iterable.clone() #can be done? > > For non-iterator iterables, it is not necessary. > For iterators in general, it is difficult at best. > For iterators returned by a constructor, it should be much easier to > re-call the constructor. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > Exactly how does itertools.tee not cover this use case? -John Graham From pyideas at rebertia.com Fri Jun 19 21:29:20 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Fri, 19 Jun 2009 12:29:20 -0700 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <4A3BD0C4.9090007@gmx.net> References: <4A3BD0C4.9090007@gmx.net> Message-ID: <50697b2c0906191229x22c31665o9a60562f1ed7db8f@mail.gmail.com> On Fri, Jun 19, 2009 at 10:54 AM, Mathias Panzenb?ck wrote: > You could write: > [x|y <- l, x <- [f(y)], x > 0] > > Oh, wait. Thats Haskell. And even in haskell you would write: > [x|x <- map f l, x > 0] > > In Python you can write: > [x for x in map(f,l) if x > 0] > > In Python 2.x you may want to write: > from itertools import imap > [x for x in imap(f,l) if x > 0] > > A more SQL like approach that would fit somewhat with pythons syntax would > be (as you can see its exactly the same lengths as the above but needs a new > name): > [f(x) as y for x in l if y > 0] > > Because in SQL you can write (IIRC): > select f(x) as y from l where y > 0; > > Maybe something like .Nets LINQ would be a nice idea to integrate in python? Comprehensions and generator expressions already give us most of the LINQ functionality. Add in `list()` and the ability to `.sort()` lists with a `key` argument and you have the entire thing, except for the one corner case being discussed. Unless I've overlooked something... Cheers, Chris -- http://blog.rebertia.com From grosser.meister.morti at gmx.net Fri Jun 19 21:56:44 2009 From: grosser.meister.morti at gmx.net (=?UTF-8?B?TWF0aGlhcyBQYW56ZW5iw7Zjaw==?=) Date: Fri, 19 Jun 2009 21:56:44 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <50697b2c0906191229x22c31665o9a60562f1ed7db8f@mail.gmail.com> References: <4A3BD0C4.9090007@gmx.net> <50697b2c0906191229x22c31665o9a60562f1ed7db8f@mail.gmail.com> Message-ID: <4A3BED7C.20002@gmx.net> Chris Rebert wrote: > On Fri, Jun 19, 2009 at 10:54 AM, Mathias > Panzenb?ck wrote: >> You could write: >> [x|y <- l, x <- [f(y)], x > 0] >> >> Oh, wait. Thats Haskell. And even in haskell you would write: >> [x|x <- map f l, x > 0] >> >> In Python you can write: >> [x for x in map(f,l) if x > 0] >> >> In Python 2.x you may want to write: >> from itertools import imap >> [x for x in imap(f,l) if x > 0] >> >> A more SQL like approach that would fit somewhat with pythons syntax would >> be (as you can see its exactly the same lengths as the above but needs a new >> name): >> [f(x) as y for x in l if y > 0] >> >> Because in SQL you can write (IIRC): >> select f(x) as y from l where y > 0; >> >> Maybe something like .Nets LINQ would be a nice idea to integrate in python? > > Comprehensions and generator expressions already give us most of the > LINQ functionality. Add in `list()` and the ability to `.sort()` lists > with a `key` argument and you have the entire thing, except for the > one corner case being discussed. Unless I've overlooked something... Yes: With LINQ its possible to build a query object out of an LINQ expression instead of evaluating it eagerly. This is used primarily to generate SQL code while still using syntax native to the host language (C#) and preserving type safety (ok the later cannot be done in python). -panzi From p.f.moore at gmail.com Fri Jun 19 22:13:17 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 19 Jun 2009 21:13:17 +0100 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <4A3BE5CB.6090505@gmail.com> References: <200906191311.12155.steve@pearwood.info> <4A3BE5CB.6090505@gmail.com> Message-ID: <79990c6b0906191313l7b67bd56v9395646041f9990@mail.gmail.com> 2009/6/19 John Graham : > Exactly how does itertools.tee not cover this use case? I've been wondering the same sort of thing. But if you're going to sequentially run through the iterator twice for i in it: do something for i in it: do something else then itertools.tee is documented as being basically equivalent to, but worse than, list(): "In general, if one iterator is going to use most or all of the data before the other iterator, it is faster to use list() instead of tee()". So the question becomes, how does list() not cover this use case. The only answer I can see is that the proposal avoids allocating temporary storage for all of the values generated. Consider xrange(1000000) - if you can call xrange twice, rather than saving a million values, you've saved a lot of memory. But this only applies when you're writing functions designed to take extremely general iterators. Someone writing a function which can take essentially arbitrary iterators, and which has to handle unboundedly large input gracefully, probably has worse problems than the lack of standard library support for something they can code themselves fairly easily... Paul. From p.f.moore at gmail.com Fri Jun 19 22:37:59 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 19 Jun 2009 21:37:59 +0100 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: Message-ID: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> 2009/6/18 Terry Reedy : > Problem: > Some functions iterate thru the input more than once. If the input is not an > iterator, there is no problem: call iter(input) again. If the input is an > iterator, that just returns the now-empty iterator. ?Bad. So: The problem is to detect which case you have. In the bad case, call list(input) and iterate over that twice. Which leaves two problems: 1. Detecting the 2 cases. 2. The cost of keeping all of the generated values in a list. For 1, you actually don't need to detect the case of a container, just call list() anyway. Unless the input is so huge that having a second copy is a substantial cost, who cares? For 2, you have 2 choices: 1. Document for callers that you'll be duplicating the input. Then your API isn't usable by people with infinite (or practically infinite) streams - but how much of a problem is this anyway? 2. Design a different API - maybe take explicit callable/arguments parameters. Then people passing a list have to do my_api(iter, L), but you're designing for huge iterators as input, so presumably you don't care about their convenience so much. Actually, I think you *can* detect that you were passed an iterable - if "iter(it) is iter(it)" returns True, then that's when you need to copy[1]. So you could document your API as copying the input if that condition is true. Then, if some user wants to use your function on a huge stream, they know what they are doing, and they can wrap their iterator in a class like your gfic (you could even document this technique in your documentation). So all of this is entirely manageable within your documentation. You could put a sample up as cookbook code, as well, I guess. And honestly, I don't believe that the problem is common enough to warrant anything more - certainly not standard library support. Paul. [1] There may be some edge cases I'm ignoring here. From pyideas at rebertia.com Fri Jun 19 23:05:49 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Fri, 19 Jun 2009 14:05:49 -0700 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <4A3BED7C.20002@gmx.net> References: <4A3BD0C4.9090007@gmx.net> <50697b2c0906191229x22c31665o9a60562f1ed7db8f@mail.gmail.com> <4A3BED7C.20002@gmx.net> Message-ID: <50697b2c0906191405p2bd47b74ga5192cdacca5c570@mail.gmail.com> On Fri, Jun 19, 2009 at 12:56 PM, Mathias Panzenb?ck wrote: > Chris Rebert wrote: >> On Fri, Jun 19, 2009 at 10:54 AM, Mathias >> Panzenb?ck wrote: >>> You could write: >>> [x|y <- l, x <- [f(y)], x > 0] >>> >>> Oh, wait. Thats Haskell. And even in haskell you would write: >>> [x|x <- map f l, x > 0] >>> >>> In Python you can write: >>> [x for x in map(f,l) if x > 0] >>> >>> In Python 2.x you may want to write: >>> from itertools import imap >>> [x for x in imap(f,l) if x > 0] >>> >>> A more SQL like approach that would fit somewhat with pythons syntax >>> would >>> be (as you can see its exactly the same lengths as the above but needs a >>> new >>> name): >>> [f(x) as y for x in l if y > 0] >>> >>> Because in SQL you can write (IIRC): >>> select f(x) as y from l where y > 0; >>> >>> Maybe something like .Nets LINQ would be a nice idea to integrate in >>> python? >> >> Comprehensions and generator expressions already give us most of the >> LINQ functionality. Add in `list()` and the ability to `.sort()` lists >> with a `key` argument and you have the entire thing, except for the >> one corner case being discussed. Unless I've overlooked something... > > Yes: With LINQ its possible to build a query object out of an LINQ > expression instead of evaluating it eagerly. This is used primarily to > generate SQL code while still using syntax native to the host language (C#) > and preserving type safety (ok the later cannot be done in python). One could probably hack that part together with lambdas, the ast module, and some black magic though. And are there any use cases besides SQL? But point taken. Cheers, Chris -- http://blog.rebertia.com From grosser.meister.morti at gmx.net Fri Jun 19 23:29:41 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Fri, 19 Jun 2009 23:29:41 +0200 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <79990c6b0906191313l7b67bd56v9395646041f9990@mail.gmail.com> References: <200906191311.12155.steve@pearwood.info> <4A3BE5CB.6090505@gmail.com> <79990c6b0906191313l7b67bd56v9395646041f9990@mail.gmail.com> Message-ID: <4A3C0345.2070901@gmx.net> Paul Moore wrote: > 2009/6/19 John Graham : >> Exactly how does itertools.tee not cover this use case? > > I've been wondering the same sort of thing. But if you're going to > sequentially run through the iterator twice > > for i in it: > do something > for i in it: > do something else > > then itertools.tee is documented as being basically equivalent to, but > worse than, list(): "In general, if one iterator is going to use most > or all of the data before the other iterator, it is faster to use > list() instead of tee()". > > So the question becomes, how does list() not cover this use case. > > The only answer I can see is that the proposal avoids allocating > temporary storage for all of the values generated. Consider > xrange(1000000) - if you can call xrange twice, rather than saving a > million values, you've saved a lot of memory. > > But this only applies when you're writing functions designed to take > extremely general iterators. Someone writing a function which can take > essentially arbitrary iterators, and which has to handle unboundedly > large input gracefully, probably has worse problems than the lack of > standard library support for something they can code themselves fairly > easily... > > Paul. I don't know if this applies here but generators can also yield infinite lists. In such an case list() is not an alternative. -panzi From lie.1296 at gmail.com Sat Jun 20 00:31:26 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sat, 20 Jun 2009 08:31:26 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <200906200035.51009.steve@pearwood.info> References: <20090618202221.GA19967@panix.com> <200906200035.51009.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > On Fri, 19 Jun 2009 05:39:32 pm Lie Ryan wrote: >> Aahz wrote: >>> On Fri, Jun 19, 2009, Lie Ryan wrote: >>>> In list/generator comprehension, currently we have no way to >>>> access the result of the expression and have to write something >>>> like this: >>>> >>>> [f(x) for x in l if f(x) > 0] >>>> >>>> if f() is heavy or non-pure (i.e. have side effects), calling f() >>>> twice might be undesirable. >>> Listcomps and genexps are like lambdas: run up against their limits >>> and you should switch to a regular for loop or generator. >> I think of it not as limitation but as an odd gap in functionality. I >> think having the semantics that the filtering is done after the >> expression part would be much more useful than the current behavior >> (filtering before expression). > > The point of the filtering is to avoid needlessly calculating a > potentially expensive expression only to throw it away. > > If you want expression first, then filter, you can get that already in a > one-liner: > > filter(lambda x: x > 0, [f(x) for x in seq]) > > Don't create new syntax when there are perfectly good functions that do > the job already. That's ugly because of the same reason for using map(): [y for y in map(lambda x: f(x), seq) if y > 0] or nested comprehension: [y for y in (f(x) for x in seq) if y > 0] From lie.1296 at gmail.com Sat Jun 20 01:45:58 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sat, 20 Jun 2009 09:45:58 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <79990c6b0906190115q20a52a85y729f51173175b045@mail.gmail.com> References: <20090618202221.GA19967@panix.com> <79990c6b0906190115q20a52a85y729f51173175b045@mail.gmail.com> Message-ID: Paul Moore wrote: > 2009/6/19 Lie Ryan : >> res = [x**x as F for x in nums if F < 100] >> (note: this is my new preferred syntax) > [...] >> Advantages of the proposal: >> - shorter >> - faster, as looping is done in C >> - more readable. The main advantage of comprehension is that it have >> standardized form, which is easier to understand, unlike a for-loop >> which can have an infinite number of variations. >> - (unnecessary) nested comprehension is an abuse. >> - with `as` keyword, no new keyword and no ambiguity since currently >> `as` cannot exist inside comprehension. >> >> Disadvantages: >> - reverses the current semantic of filtering-then-expression. This >> shouldn't be too much problem since side-effect on the expression part >> is a cardinal sin and... >> - if the expression part is heavy, it might be possible to do >> optimization by filtering first when the filter part does not require >> the result (i.e. when there is no "as" clause). A good side effect of >> this optimization is codes that relies on filtering being done before >> expression will just work as they cannot contain an `as` keyword. >> (As "simple is better than complex", I actually don't really like `as` >> can change evaluation order; I much prefer to keep everything simple and >> consistent, i.e. always evaluate expression then filter or otherwise) >> >> possible syntaxes: >> - [x**x as F for x in nums if F < 100] >> the as keyword is already often used to rename things >> (in with, import, etc) I like this one much better than @. The as >> part, of course, is optional >> - [x**x for x in nums if @ < 100] >> the initial proposed syntax, ugly as hell. > > OK, with this explanation (and the new syntax) I see what you're > getting at better. > > However, changing the order of evaluate vs filter is a huge > compatibility problem. There's no way this will be possible. Even with > syntax triggering the change (so that it's one way with the "as", the > other without), that's a disaster waiting to happen. How about this syntax which would solve your concern for the semantic change: [x**x as F for x in lst if F() < 100] it's similar to original `as` proposal, except that F is a callable instead of direct value. The advantage of F being callable is that it does not need semantic change, the filtering part will be done before expression just like it is right now. However, we can explicitly request for the expression to be evaluated by calling F(); the return value of F() will be saved and reused for the final result and other calls to F(). A diagrammatic explanation: +--------------------------- | this is the part that name | the expression's callable | --+- [x**x as F for x in lst if x and F() < 100 and isvalid(F())] -+-- ---+---------------------------- | | -+- -+- | the filtering part is | | | | evaluated before | | | | expression just like | | | | current behavior | | | | -----------------------+ | | | | | | then when F gets called; | | | expression is evaluated, | | | cached, and returned | | | ---------------------------+ | | | | F is called again, return cached result | | ----------------------------------------+ | | at the end of the day, if F is called, | return the cached result, else evaluate | the expression and use that +----------------------------------------- using the as-callable syntax, the semantic of this: [f(x) as F for x in lst if g(F())] would be similar to: result = [] for x in lst: # F() ensures f(x) will be only ever be called once def F(): nonlocal _cache if not _cache: _cache = f(x) return _cache _cache = None if g(F()): result.append(F()) the only disadvantage of this as-callable is if you forgot to call F. > You have at least 3 explicit ways of stating your intent (genexp > inside listcomp, map inside listcomp, explicit loop). None is as > clean-looking as your (amended) proposal, but they work now, and they > don't have the semantic issues of your proposal. > > (For a more general, more radical, equally certain to be shot down, > option, which at least doesn't introduce the change in semantics, you > could try proposing "as" as an assignment-as-expression operator. So > you could have > > [y for x in l if (f(x) as y) < 100] > > Hmm, on second thoughts - no, don't bother... :-)) It took me several minutes to understand that one... and no, that syntax as makes it way too easy to be too creative in list comprehension, devaluing the "standard form" which IMO is the strongest point of list comprehension. Not to mention that that syntax moved the expression part to be inside the filtering part... which is quite... disturbing... From steve at pearwood.info Sat Jun 20 02:32:51 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Jun 2009 10:32:51 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> Message-ID: <200906201032.52123.steve@pearwood.info> On Sat, 20 Jun 2009 08:31:26 am Lie Ryan wrote: > > If you want expression first, then filter, you can get that already > > in a one-liner: > > > > filter(lambda x: x > 0, [f(x) for x in seq]) > > > > Don't create new syntax when there are perfectly good functions > > that do the job already. > > That's ugly because of the same reason for using map(): > [y for y in map(lambda x: f(x), seq) if y > 0] You don't like lambda? Fine, define an external function first. Then you can write: filter(pred, (f(x) for x in seq)) There's no violation of DRY, there's no redundancy, there's no lambda, there's no "y" variable needed. What's ugly about it? > or nested comprehension: > [y for y in (f(x) for x in seq) if y > 0] You seem to be labouring under the misapprehension that anything that requires two steps instead of one is "ugly". I find the nested comprehension perfectly readable, although for more complicated cases I'd split it into two explicit steps. It is (almost) completely general, covering both filtering on *both* input args and output args: gen = (3*x**2-5*x+4 for x in seq if x % 3 != 2) result = [y for y in gen if -3 < y < 3] The only case it doesn't cover is where the second filter depends on the value of x, and even that can be covered with a *tiny* bit more work: gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) result = [y[1] for y in gen if -y[0] < y[1] < y[0]] It requires no new syntax, no changes to the behaviour of list comps, no new meaning on "as", it's understandable and readable. Compare your suggestion: [3*x**2-5*x+4 as y for x in seq if (x % 3 != 2) and (-x < y < x)] Disadvantages: - It requires new syntax. - It requires new behaviour to list comps and generator expressions. - It creates yet another meaning to the keyword "as". Advantages: - It requires no intermediate tuple. But since intermediate tuples are only required for a tiny proportion of use-cases, this is not much of a advantage. - It loops over the data once rather than twice, but since it does twice as much work inside the loop the only saving is the setup and teardown costs of the second loop. Truly a micro-optimization, and premature at that. -- Steven D'Aprano From ben+python at benfinney.id.au Sat Jun 20 04:45:01 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 20 Jun 2009 12:45:01 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional References: <20090618202221.GA19967@panix.com> Message-ID: <8763er1v4i.fsf@benfinney.id.au> Jim Jewett writes: > On Fri, Jun 19, 2009 at 3:39 AM, Lie Ryan wrote: > > > res = [x**x as F for x in nums if F < 100] > > This, I have wanted. You have it: res = [f for f in (x**x for x in nums) if f < 100] In addition to the fact that this works now in existing Python, I find it clearer than the above syntax you say you want. -- \ ?It's my belief we developed language because of our deep inner | `\ need to complain.? ?Jane Wagner, via Lily Tomlin | _o__) | Ben Finney From john.a.graham at gmail.com Sat Jun 20 05:55:42 2009 From: john.a.graham at gmail.com (John Graham) Date: Fri, 19 Jun 2009 22:55:42 -0500 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <4A3C0345.2070901@gmx.net> References: <200906191311.12155.steve@pearwood.info> <4A3BE5CB.6090505@gmail.com> <79990c6b0906191313l7b67bd56v9395646041f9990@mail.gmail.com> <4A3C0345.2070901@gmx.net> Message-ID: <4A3C5DBE.8060008@gmail.com> Mathias Panzenb?ck wrote: > Paul Moore wrote: >> 2009/6/19 John Graham : >>> Exactly how does itertools.tee not cover this use case? >> >> I've been wondering the same sort of thing. But if you're going to >> sequentially run through the iterator twice >> >> for i in it: >> do something >> for i in it: >> do something else >> >> then itertools.tee is documented as being basically equivalent to, but >> worse than, list(): "In general, if one iterator is going to use most >> or all of the data before the other iterator, it is faster to use >> list() instead of tee()". >> >> So the question becomes, how does list() not cover this use case. >> >> The only answer I can see is that the proposal avoids allocating >> temporary storage for all of the values generated. Consider >> xrange(1000000) - if you can call xrange twice, rather than saving a >> million values, you've saved a lot of memory. >> >> But this only applies when you're writing functions designed to take >> extremely general iterators. Someone writing a function which can take >> essentially arbitrary iterators, and which has to handle unboundedly >> large input gracefully, probably has worse problems than the lack of >> standard library support for something they can code themselves fairly >> easily... >> >> Paul. > > I don't know if this applies here but generators can also yield > infinite lists. In such an case list() is not an alternative. > > -panzi > Hrm, that leads to the question, is it mathematically possible to iterate over an infinite list... twice? :) -John From greg.ewing at canterbury.ac.nz Sat Jun 20 07:59:51 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 20 Jun 2009 17:59:51 +1200 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <4A3C5DBE.8060008@gmail.com> References: <200906191311.12155.steve@pearwood.info> <4A3BE5CB.6090505@gmail.com> <79990c6b0906191313l7b67bd56v9395646041f9990@mail.gmail.com> <4A3C0345.2070901@gmx.net> <4A3C5DBE.8060008@gmail.com> Message-ID: <4A3C7AD7.5000607@canterbury.ac.nz> John Graham wrote: > Hrm, that leads to the question, is it mathematically possible to > iterate over an infinite list... twice? :) Sure, as long as you don't insist on finishing the first iteration before starting the second one... And it takes exactly the same amount of time as doing it once. :-) -- Greg From stephen at xemacs.org Sat Jun 20 08:13:24 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 20 Jun 2009 15:13:24 +0900 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <4A3C5DBE.8060008@gmail.com> References: <200906191311.12155.steve@pearwood.info> <4A3BE5CB.6090505@gmail.com> <79990c6b0906191313l7b67bd56v9395646041f9990@mail.gmail.com> <4A3C0345.2070901@gmx.net> <4A3C5DBE.8060008@gmail.com> Message-ID: <87ab43juuz.fsf@uwakimon.sk.tsukuba.ac.jp> John Graham writes: > Hrm, that leads to the question, is it mathematically possible to > iterate over an infinite list... twice? :) Of course. Some infinite ordinals are bigger than others, you know. :) From g.brandl at gmx.net Sat Jun 20 10:13:06 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 20 Jun 2009 10:13:06 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <20090618202221.GA19967@panix.com> <200906200035.51009.steve@pearwood.info> Message-ID: Lie Ryan schrieb: > Steven D'Aprano wrote: >> On Fri, 19 Jun 2009 05:39:32 pm Lie Ryan wrote: >>> Aahz wrote: >>>> On Fri, Jun 19, 2009, Lie Ryan wrote: >>>>> In list/generator comprehension, currently we have no way to >>>>> access the result of the expression and have to write something >>>>> like this: >>>>> >>>>> [f(x) for x in l if f(x) > 0] >>>>> >>>>> if f() is heavy or non-pure (i.e. have side effects), calling f() >>>>> twice might be undesirable. >>>> Listcomps and genexps are like lambdas: run up against their limits >>>> and you should switch to a regular for loop or generator. >>> I think of it not as limitation but as an odd gap in functionality. I >>> think having the semantics that the filtering is done after the >>> expression part would be much more useful than the current behavior >>> (filtering before expression). >> >> The point of the filtering is to avoid needlessly calculating a >> potentially expensive expression only to throw it away. >> >> If you want expression first, then filter, you can get that already in a >> one-liner: >> >> filter(lambda x: x > 0, [f(x) for x in seq]) >> >> Don't create new syntax when there are perfectly good functions that do >> the job already. > > That's ugly because of the same reason for using map(): > [y for y in map(lambda x: f(x), seq) if y > 0] Especially if f is already a handy callable of one argument, no need to use a lambda. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From steve at pearwood.info Sat Jun 20 11:09:40 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Jun 2009 19:09:40 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <79990c6b0906190115q20a52a85y729f51173175b045@mail.gmail.com> Message-ID: <200906201909.41140.steve@pearwood.info> On Sat, 20 Jun 2009 09:45:58 am Lie Ryan wrote: > How about this syntax which would solve your concern for the semantic > change: > > [x**x as F for x in lst if F() < 100] Let's look at what that would be equivalent to. L = [] for x in lst: F = lambda x=x: x**x # Need to use default value in the # lambda otherwise all elements will have the same value. tmp = F() if tmp < 100: L.append(tmp) It's not clear why you think this is an improvement over: [x**x as F for x in lst if F < 100] # note the missing ()s which would be equivalent to: L = [] for x in lst: F = x**x if F < 100: L.append(F) Despite what you say here: > The advantage of F being callable is that it does not need semantic > change, the filtering part will be done before expression just like > it is right now. That's not true. It can't be true. If you want to filter on x**x being greater than 100, you need to calculate x**x first. In theory, a sufficiently clever compiler could recognise that, say, x**x < 100 implies 0 <= x < 3.59728 (approx), but in general, you can't predict the value of f(x) without actually calculating f(x). What happens if you accidentally forget to put brackets after the F expression? Do you get a syntax error? Undefined behaviour? A runtime exception? -- Steven D'Aprano From steve at pearwood.info Sat Jun 20 11:24:12 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 20 Jun 2009 19:24:12 +1000 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <4A3C5DBE.8060008@gmail.com> References: <4A3C0345.2070901@gmx.net> <4A3C5DBE.8060008@gmail.com> Message-ID: <200906201924.12886.steve@pearwood.info> On Sat, 20 Jun 2009 01:55:42 pm John Graham wrote: > Hrm, that leads to the question, is it mathematically possible to > iterate over an infinite list... twice? :) Mathematically, yes. Practically, no. Mathematically, the trick is to ensure that the time it takes to iterate over each item approaches zero sufficiently fast. E.g. if it takes 1 unit of time to process the first item, and 1/2 units of time for the second, and 1/4 for the third, 1/8 for the fourth, etc, then you can iterate over an infinite number of items in 2 units of time total. Then you can do them all again :) -- Steven D'Aprano From python at mrabarnett.plus.com Sat Jun 20 15:22:30 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 20 Jun 2009 14:22:30 +0100 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <8763er1v4i.fsf@benfinney.id.au> References: <20090618202221.GA19967@panix.com> <8763er1v4i.fsf@benfinney.id.au> Message-ID: <4A3CE296.5040407@mrabarnett.plus.com> Ben Finney wrote: > Jim Jewett writes: > >> On Fri, Jun 19, 2009 at 3:39 AM, Lie Ryan wrote: >> >>> res = [x**x as F for x in nums if F < 100] >> This, I have wanted. > > You have it: > > res = [f for f in (x**x for x in nums) if f < 100] > > In addition to the fact that this works now in existing Python, I find > it clearer than the above syntax you say you want. > How about: res = [F for x in nums with x**x as F if F < 100] :-) From ironfroggy at gmail.com Sat Jun 20 15:25:33 2009 From: ironfroggy at gmail.com (Calvin Spealman) Date: Sat, 20 Jun 2009 09:25:33 -0400 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <4A3CE296.5040407@mrabarnett.plus.com> References: <20090618202221.GA19967@panix.com> <8763er1v4i.fsf@benfinney.id.au> <4A3CE296.5040407@mrabarnett.plus.com> Message-ID: <76fd5acf0906200625l180e81c3l5e2cbab4474911b6@mail.gmail.com> On Sat, Jun 20, 2009 at 9:22 AM, MRAB wrote: > Ben Finney wrote: >> >> Jim Jewett writes: >> >>> On Fri, Jun 19, 2009 at 3:39 AM, Lie Ryan wrote: >>> >>>> res = [x**x as F for x in nums if F < 100] >>> >>> This, I have wanted. >> >> You have it: >> >> ? ?res = [f for f in (x**x for x in nums) if f < 100] >> >> In addition to the fact that this works now in existing Python, I find >> it clearer than the above syntax you say you want. >> > How about: > > ? res = [F for x in nums with x**x as F if F < 100] > > :-) That toggles the first part of the comprehensions to be or not be an expression, depending on if there is a with clause later. You could miss this when you read it, and it opens the door to doing more strange things, like: res = [F/2 for x in nums with x**x as F if F < 100] This is basically a strangely syntaxed nested loop -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy From python at mrabarnett.plus.com Sat Jun 20 16:16:16 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 20 Jun 2009 15:16:16 +0100 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <76fd5acf0906200625l180e81c3l5e2cbab4474911b6@mail.gmail.com> References: <20090618202221.GA19967@panix.com> <8763er1v4i.fsf@benfinney.id.au> <4A3CE296.5040407@mrabarnett.plus.com> <76fd5acf0906200625l180e81c3l5e2cbab4474911b6@mail.gmail.com> Message-ID: <4A3CEF30.20305@mrabarnett.plus.com> Calvin Spealman wrote: > On Sat, Jun 20, 2009 at 9:22 AM, MRAB wrote: >> Ben Finney wrote: >>> Jim Jewett writes: >>> >>>> On Fri, Jun 19, 2009 at 3:39 AM, Lie Ryan wrote: >>>> >>>>> res = [x**x as F for x in nums if F < 100] >>>> This, I have wanted. >>> You have it: >>> >>> res = [f for f in (x**x for x in nums) if f < 100] >>> >>> In addition to the fact that this works now in existing Python, I find >>> it clearer than the above syntax you say you want. >>> >> How about: >> >> res = [F for x in nums with x**x as F if F < 100] >> >> :-) > > That toggles the first part of the comprehensions to be or not be an > expression, depending on if there is a with clause later. You could > miss this when you read it, and it opens the door to doing more > strange things, like: > > res = [F/2 for x in nums with x**x as F if F < 100] > > This is basically a strangely syntaxed nested loop > I don't know what you mean. >>> def F(m, x): print m, x return x >>> [foo("value", x) for x in range(5) if foo("test", x) % 2] test 0 test 1 value 1 test 2 test 3 value 3 test 4 [1, 3] The test is done first, so it's equivalent to: >>> results = [] >>> for x in range(5): if foo("test", x) % 2: results.append(foo("value", x)) test 0 test 1 value 1 test 2 test 3 value 3 test 4 That means that: res = [x**x for x in nums if x**x < 100] is equivalent to: res = [] for x in nums : if x**x < 100: res.append(x**x) My suggestions turns: res = [] for x in nums : F = x**x # <== temporary variable if F < 100: res.append(F) into: res = [F for x in nums with x**x as F if F < 100] ^^^^^^^^^^^^^^ temporary variable From grosser.meister.morti at gmx.net Sat Jun 20 16:56:14 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sat, 20 Jun 2009 16:56:14 +0200 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <4A3C7AD7.5000607@canterbury.ac.nz> References: <200906191311.12155.steve@pearwood.info> <4A3BE5CB.6090505@gmail.com> <79990c6b0906191313l7b67bd56v9395646041f9990@mail.gmail.com> <4A3C0345.2070901@gmx.net> <4A3C5DBE.8060008@gmail.com> <4A3C7AD7.5000607@canterbury.ac.nz> Message-ID: <4A3CF88E.5080903@gmx.net> Greg Ewing wrote: > John Graham wrote: > >> Hrm, that leads to the question, is it mathematically possible to >> iterate over an infinite list... twice? :) > > Sure, as long as you don't insist on finishing the > first iteration before starting the second one... > > And it takes exactly the same amount of time as > doing it once. :-) > Thats what I was thinking about. Like: def foo(seq): for x, y, z in izip(seq, islice(seq,1,None,2), islice(seq,2,None,3)): ... From lie.1296 at gmail.com Sat Jun 20 19:58:22 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 03:58:22 +1000 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <200906201924.12886.steve@pearwood.info> References: <4A3C0345.2070901@gmx.net> <4A3C5DBE.8060008@gmail.com> <200906201924.12886.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > On Sat, 20 Jun 2009 01:55:42 pm John Graham wrote: > >> Hrm, that leads to the question, is it mathematically possible to >> iterate over an infinite list... twice? :) > > Mathematically, yes. Practically, no. > > Mathematically, the trick is to ensure that the time it takes to iterate > over each item approaches zero sufficiently fast. E.g. if it takes 1 > unit of time to process the first item, and 1/2 units of time for the > second, and 1/4 for the third, 1/8 for the fourth, etc, then you can > iterate over an infinite number of items in 2 units of time total. Then > you can do them all again :) It is also possible to do parallel iteration, but I'm not sure that mathematically parallel processing makes sense. I'm not even sure that mathematically there is such thing as iteration (that does not involve recursion). From lie.1296 at gmail.com Sat Jun 20 20:08:05 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 04:08:05 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <200906201909.41140.steve@pearwood.info> References: <79990c6b0906190115q20a52a85y729f51173175b045@mail.gmail.com> <200906201909.41140.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: >> The advantage of F being callable is that it does not need semantic >> change, the filtering part will be done before expression just like >> it is right now. > > That's not true. It can't be true. If you want to filter on x**x being > greater than 100, you need to calculate x**x first. In theory, a > sufficiently clever compiler could recognise that, say, x**x < 100 > implies 0 <= x < 3.59728 (approx), but in general, you can't predict > the value of f(x) without actually calculating f(x). Did you read the middle part of the post and the diagram, which address the question you're asking and how it would be handled? > What happens if you accidentally forget to put brackets after the F > expression? Do you get a syntax error? Undefined behaviour? A runtime > exception? As F is just callable, F > 100 should result in comparison of number against callable. But as it is rare that you really actually wanted to do such thing, I think python can also be a little protective and issue a warning. From dreamingforward at gmail.com Sat Jun 20 20:11:49 2009 From: dreamingforward at gmail.com (average) Date: Sat, 20 Jun 2009 11:11:49 -0700 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional Message-ID: <913f9f570906201111r39c615cfsd341356197889110@mail.gmail.com> From: Lie Ryan wrote: > In list/generator comprehension, currently we have no way to access the > result of the expression and have to write something like this: > > [f(x) for x in l if f(x) > 0] > > if f() is heavy or non-pure (i.e. have side effects), calling f() twice > might be undesirable. > > I'm suggesting about something like: > > [f(x) for x in l if @ > 0] > > where @ is the result of the listcomp's expression (i.e. f(x)) > > Personally, I don't like the use of symbols like @, as python is not > perl. I'm still thinking of a better syntax/approach and is open for > suggestion. > > What do you guys think? I think it's a fine idea. The English language has evolved such a construct, typically used when one wants to highlight (rather than simply *refer* (as a pronoun does) to) the dominant noun without resorting to redundancy. [f(x) for x in l if same > 0] OR another possibility [f(x) for x in l if self > 0] obviously syntax highlighting makes this (or any keyword) standout better. Would be interesting to consider other usages for such a keyword addition (in the case of the former). Aahz: > Listcomps and genexps are like lambdas: run up against their limits and > you should switch to a regular for loop or generator. Generally decent advice, but there is value to exploring the Kolomogorov optimums for a given expressive need (compressing the highest amount of meaning in the simplest construction). In fact, I consider Python to be one of best languages in this regard (it's syntax being the evolution of programming-expressive compactness while being complete in itself. (In contrast to Perl, for example, which offloads the complexity to the programmer.) mark From lie.1296 at gmail.com Sat Jun 20 20:26:57 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 04:26:57 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <200906201032.52123.steve@pearwood.info> References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > The only case it doesn't cover is where the second filter depends on the > value of x, and even that can be covered with a *tiny* bit more work: > > gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) > result = [y[1] for y in gen if -y[0] < y[1] < y[0]] > > It requires no new syntax, no changes to the behaviour of list comps, no > new meaning on "as", it's understandable and readable. > > Compare your suggestion: > > [3*x**2-5*x+4 as y for x in seq if (x % 3 != 2) and (-x < y < x)] For me, this one is much clearer, understandable, and readable than y[0] and y[1]; and you still have the option to split them if you think y[0] and y[1] is better. > Disadvantages: > > - It requires new syntax. > - It requires new behaviour to list comps and generator expressions. > - It creates yet another meaning to the keyword "as". "as" is already used for renaming keyword in "with" and "import" statement, so I don't think it actually creates any more meaning that we don't already have. From lie.1296 at gmail.com Sat Jun 20 20:35:21 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 04:35:21 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <20090618202221.GA19967@panix.com> <200906200035.51009.steve@pearwood.info> Message-ID: Georg Brandl wrote: > Lie Ryan schrieb: >> That's ugly because of the same reason for using map(): >> [y for y in map(lambda x: f(x), seq) if y > 0] > > Especially if f is already a handy callable of one argument, no need to use > a lambda. Ahh... yes. Sometimes I actually meant f(x) to be a substitute to any expression (both simple: x**2 or complex: foo(x)) but well, I didn't actually see that one... From tjreedy at udel.edu Sat Jun 20 21:07:23 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 20 Jun 2009 15:07:23 -0400 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <200906201032.52123.steve@pearwood.info> References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > On Sat, 20 Jun 2009 08:31:26 am Lie Ryan wrote: > You don't like lambda? Fine, define an external function first. Then you > can write: > > filter(pred, (f(x) for x in seq)) > > There's no violation of DRY, there's no redundancy, there's no lambda, > there's no "y" variable needed. What's ugly about it? I think its great and that it kills any justification for the proposal. tjr From grosser.meister.morti at gmx.net Sat Jun 20 21:23:54 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sat, 20 Jun 2009 21:23:54 +0200 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <4A3C0345.2070901@gmx.net> <4A3C5DBE.8060008@gmail.com> <200906201924.12886.steve@pearwood.info> Message-ID: <4A3D374A.5080709@gmx.net> Lie Ryan wrote: > Steven D'Aprano wrote: >> On Sat, 20 Jun 2009 01:55:42 pm John Graham wrote: >> >>> Hrm, that leads to the question, is it mathematically possible to >>> iterate over an infinite list... twice? :) >> Mathematically, yes. Practically, no. >> >> Mathematically, the trick is to ensure that the time it takes to iterate >> over each item approaches zero sufficiently fast. E.g. if it takes 1 >> unit of time to process the first item, and 1/2 units of time for the >> second, and 1/4 for the third, 1/8 for the fourth, etc, then you can >> iterate over an infinite number of items in 2 units of time total. Then >> you can do them all again :) > > It is also possible to do parallel iteration, but I'm not sure that > mathematically parallel processing makes sense. I'm not even sure that > mathematically there is such thing as iteration (that does not involve > recursion). > Well there is something called "process algebra" (at least I think its called that way, in german its: Prozess Algebra). This provides a (mathematical?) model for such things. From grosser.meister.morti at gmx.net Sat Jun 20 21:29:27 2009 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sat, 20 Jun 2009 21:29:27 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <913f9f570906201111r39c615cfsd341356197889110@mail.gmail.com> References: <913f9f570906201111r39c615cfsd341356197889110@mail.gmail.com> Message-ID: <4A3D3897.7010509@gmx.net> average wrote: > From: Lie Ryan wrote: >> In list/generator comprehension, currently we have no way to access the >> result of the expression and have to write something like this: >> >> [f(x) for x in l if f(x) > 0] >> >> if f() is heavy or non-pure (i.e. have side effects), calling f() twice >> might be undesirable. >> >> I'm suggesting about something like: >> >> [f(x) for x in l if @ > 0] >> >> where @ is the result of the listcomp's expression (i.e. f(x)) >> >> Personally, I don't like the use of symbols like @, as python is not >> perl. I'm still thinking of a better syntax/approach and is open for >> suggestion. >> >> What do you guys think? > > I think it's a fine idea. The English language has evolved such a > construct, typically used when one wants to highlight (rather than > simply *refer* (as a pronoun does) to) the dominant noun without > resorting to redundancy. > > [f(x) for x in l if same > 0] OR another possibility > [f(x) for x in l if self > 0] > > obviously syntax highlighting makes this (or any keyword) standout > better. Would be interesting to consider other usages for such a > keyword addition (in the case of the former). > I dunno. What about nested constructs? Such unnamed variables lead to ambiguities. E.g.: [f(x) for x in l1 if y > sum(z for f in l2 if @ > 0)] To which value does the @ refer to? To the most inner one would be an obvious solution. But then, how do you refer to the outer value? It suddenly gets invisible (overwritten?). Use @@ for the next outer value? No I don't like it. -panzi From lie.1296 at gmail.com Sat Jun 20 21:51:09 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 05:51:09 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: Terry Reedy wrote: > Steven D'Aprano wrote: >> On Sat, 20 Jun 2009 08:31:26 am Lie Ryan wrote: > >> You don't like lambda? Fine, define an external function first. Then >> you can write: >> >> filter(pred, (f(x) for x in seq)) >> >> There's no violation of DRY, there's no redundancy, there's no lambda, >> there's no "y" variable needed. What's ugly about it? > > I think its great and that it kills any justification for the proposal. > > tjr I hate it. It mixes map/filter style and comprehension style; and the fact it does so in a single line only makes it worse. Not that it would be any better in two lines. From tjreedy at udel.edu Sat Jun 20 21:54:59 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 20 Jun 2009 15:54:59 -0400 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> Message-ID: Paul Moore wrote: > 2009/6/18 Terry Reedy : >> Problem: >> Some functions iterate thru the input more than once. If the input is not an >> iterator, there is no problem: call iter(input) again. If the input is an >> iterator, that just returns the now-empty iterator. Bad. > > So: The problem is to detect which case you have. In the bad case, > call list(input) and iterate over that twice. > Which leaves two problems: > 1. Detecting the 2 cases. > 2. The cost of keeping all of the generated values in a list. > > For 1, you actually don't need to detect the case of a container, just > call list() anyway. Unless the input is so huge that having a second > copy is a substantial cost, who cares? > > For 2, you have 2 choices: > > 1. Document for callers that you'll be duplicating the input. Then > your API isn't usable by people with infinite (or practically > infinite) streams - but how much of a problem is this anyway? > > 2. Design a different API - maybe take explicit callable/arguments > parameters. Then people passing a list have to do my_api(iter, L), but > you're designing for huge iterators as input, so presumably you don't > care about their convenience so much. > > Actually, I think you *can* detect that you were passed an iterable - > if "iter(it) is iter(it)" returns True, then that's when you need to > copy[1]. So you could document your API as copying the input if that > condition is true. Then, if some user wants to use your function on a > huge stream, they know what they are doing, and they can wrap their > iterator in a class like your gfic (you could even document this > technique in your documentation). > > So all of this is entirely manageable within your documentation. You > could put a sample up as cookbook code, as well, I guess. And You are looking at the issue from the callee's standpoint. And you have made some good suggestions from that viewpoint. I am looking at the issue from the caller's viewpoint. I have an iterator constructor and I want to use an existing function that requires a re-iterable. > honestly, I don't believe that the problem is common enough to warrant > anything more - certainly not standard library support. The issue is more frequent in Py3, which largely shifts from lists to iterators as the common sequence interchange format. Py3 is what I use and the target of my suggestion. I should have said that. Map and filter, for instance, are now iterator classes, which is to say, iterator constructors, rather than list-returning functions. A call such as somefunc(map(f,l)) which works fine in 2.x will not work in 3.x if somefunc requires a re-iterable. But I agree that it seems to still be too rare for stdlib. I will, however, put the class in the code part of my book-in-progress as 'reiterable'. I am sure that the discussion here will help improve the text discussion of this issue and this class. Terry Jan Reedy From g.brandl at gmx.net Sat Jun 20 22:00:42 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 20 Jun 2009 22:00:42 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: Lie Ryan schrieb: > Terry Reedy wrote: >> Steven D'Aprano wrote: >>> On Sat, 20 Jun 2009 08:31:26 am Lie Ryan wrote: >> >>> You don't like lambda? Fine, define an external function first. Then >>> you can write: >>> >>> filter(pred, (f(x) for x in seq)) >>> >>> There's no violation of DRY, there's no redundancy, there's no lambda, >>> there's no "y" variable needed. What's ugly about it? >> >> I think its great and that it kills any justification for the proposal. >> >> tjr > > I hate it. It mixes map/filter style and comprehension style; and the > fact it does so in a single line only makes it worse. Not that it would > be any better in two lines. Taking this further, using only map/filter style like this filter(pred, map(f, seq)) takes two steps. Why is it so bad that doing it in a listcomp (el for el in (f(y) for y in seq) if el > 2) takes two steps as well? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From lie.1296 at gmail.com Sat Jun 20 22:29:28 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 06:29:28 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: Georg Brandl wrote: > Lie Ryan schrieb: >> Terry Reedy wrote: >>> Steven D'Aprano wrote: >>>> On Sat, 20 Jun 2009 08:31:26 am Lie Ryan wrote: >>>> You don't like lambda? Fine, define an external function first. Then >>>> you can write: >>>> >>>> filter(pred, (f(x) for x in seq)) >>>> >>>> There's no violation of DRY, there's no redundancy, there's no lambda, >>>> there's no "y" variable needed. What's ugly about it? >>> I think its great and that it kills any justification for the proposal. >>> >>> tjr >> I hate it. It mixes map/filter style and comprehension style; and the >> fact it does so in a single line only makes it worse. Not that it would >> be any better in two lines. > > Taking this further, using only map/filter style like this > > filter(pred, map(f, seq)) > > takes two steps. Why is it so bad that doing it in a listcomp > > (el for el in (f(y) for y in seq) if el > 2) > > takes two steps as well? > > Georg > Comprehension, by its nature, is map+filter in a single expression. Nested comprehension is (map+filter)+(map+filter). (where + is some sort of function composition) The proposal enables comprehension to become filter+map+filter, map+filter, filter+map, or map-only; eliminating the redundant map in (map+filter)+(map+filter). The filter() and map() functions are two separate function in the first place, so there is no redundancy in it. Mixing the two styles is ugly since it means you have to think in two separate (though related) paradigms. From p.f.moore at gmail.com Sun Jun 21 00:16:55 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 20 Jun 2009 23:16:55 +0100 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> Message-ID: <79990c6b0906201516g5554d1damd0cf3c058248fd6e@mail.gmail.com> 2009/6/20 Terry Reedy : > I am looking at the issue from the caller's viewpoint. > I have an iterator constructor and I want to use an existing function that > requires a re-iterable. OK. But how many functions require a re-iterable? It doesn't seem to me to be a common requirement. And if they did, how often would it be a problem to construct a list? I see your point, but I can't imagine it's a common need. But your experience may differ. Paul. From ncoghlan at gmail.com Sun Jun 21 00:31:44 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Jun 2009 08:31:44 +1000 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <79990c6b0906201516g5554d1damd0cf3c058248fd6e@mail.gmail.com> References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> <79990c6b0906201516g5554d1damd0cf3c058248fd6e@mail.gmail.com> Message-ID: <4A3D6350.8060004@gmail.com> Paul Moore wrote: > 2009/6/20 Terry Reedy : >> I am looking at the issue from the caller's viewpoint. >> I have an iterator constructor and I want to use an existing function that >> requires a re-iterable. > > OK. But how many functions require a re-iterable? It doesn't seem to > me to be a common requirement. > > And if they did, how often would it be a problem to construct a list? > > I see your point, but I can't imagine it's a common need. But your > experience may differ. Mostly avoiding this discussion, but I think the use cases are even narrower than that. 1. I've never seen a function parameter spec'ed as iterable vs reiterable. It's always iterable vs sequence. 2. "Sequence" implies a lot more than "reiterable" does: it also implies length, containment test support, random access. So any idea focused solely on reiterability misses out on those extra features. 3. Terry's idea also assumes a deterministic data source. An iterator with a random element or a data feed from the real world can't be used with it (because the original sequence can't be reproduced without saving it). 4. The only reliable way to make an arbitrary iterator reiterable is to cache the results in memory, but we already have a way to do that (i.e. store it in a list or tuple) 5. For iterator based cases where only *some* of the values need to be kept around rather than all of them, then the algorithm implementation should be redesigned so it can make effective use of itertools.tee() In other words, this strikes as a fairly complicated solution to a rather limited problem. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ben+python at benfinney.id.au Sun Jun 21 01:35:57 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 21 Jun 2009 09:35:57 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: <87d48yzdeq.fsf@benfinney.id.au> Lie Ryan writes: > Steven D'Aprano wrote: > > > The only case it doesn't cover is where the second filter depends on > > the value of x, and even that can be covered with a *tiny* bit more > > work: > > > > gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) > > result = [y[1] for y in gen if -y[0] < y[1] < y[0]] > > > > It requires no new syntax, no changes to the behaviour of list > > comps, no new meaning on "as", it's understandable and readable. I think it would be more readable without index references, but instead using tuple unpacking:: gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) result = [b for (a, b) in gen if -a < b < a] It can even be done as a single expression without (IMO) significantly affecting readability:: result = [ b for (a, b) in ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) if -a < b < a] > > Compare your suggestion: > > > > [3*x**2-5*x+4 as y for x in seq if (x % 3 != 2) and (-x < y < x)] > > For me, this one is much clearer, understandable, and readable than > y[0] and y[1]; and you still have the option to split them if you > think y[0] and y[1] is better. I hope you'll agree that my above suggestion retains this, without needing any new syntax. -- \ ?If you go to a costume party at your boss's house, wouldn't | `\ you think a good costume would be to dress up like the boss's | _o__) wife? Trust me, it's not.? ?Jack Handey | Ben Finney From tjreedy at udel.edu Sun Jun 21 02:21:40 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 20 Jun 2009 20:21:40 -0400 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <4A3D6350.8060004@gmail.com> References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> <79990c6b0906201516g5554d1damd0cf3c058248fd6e@mail.gmail.com> <4A3D6350.8060004@gmail.com> Message-ID: Nick Coghlan wrote: > Paul Moore wrote: >> 2009/6/20 Terry Reedy : >>> I am looking at the issue from the caller's viewpoint. >>> I have an iterator constructor and I want to use an existing function that >>> requires a re-iterable. >> OK. But how many functions require a re-iterable? It doesn't seem to >> me to be a common requirement. >> >> And if they did, how often would it be a problem to construct a list? >> >> I see your point, but I can't imagine it's a common need. But your >> experience may differ. > > Mostly avoiding this discussion, but I think the use cases are even > narrower than that. > > 1. I've never seen a function parameter spec'ed as iterable vs > reiterable. It's always iterable vs sequence. The latter could be over-spec'ed because people are used to that being the choice, or because 'sequence' is being used as a synonym for 'reiterable' even though it implies more than is needed. > 2. "Sequence" implies a lot more than "reiterable" does: it also implies > length, containment test support, random access. So any idea focused > solely on reiterability misses out on those extra features. Which may or may not be needed. def pairs(finite_reiterable) for i in finite_reiterable: for j in finite)reiterable: return (i,j) is a simple function for which 'finite_reiterable' is exactly the appropriate spec. The issue is whether it is best to wrap it internally (inline) with a generic list(it) or to let the caller special-case iterator constructors. > 3. Terry's idea also assumes a deterministic data source. An iterator > with a random element or a data feed from the real world can't be used > with it (because the original sequence can't be reproduced without > saving it). Yes. > 4. The only reliable way to make an arbitrary iterator reiterable is to > cache the results in memory, but we already have a way to do that (i.e. > store it in a list or tuple) Which I mentioned in my original post. I specificlly considered how to make iterator constructors, more common in 3.x, reiterable *without* storing everything in memory. Avoiding such storage is why map, filter, range, for example, were changed. So it hardly seems like an oddball aim. > 5. For iterator based cases where only *some* of the values need to be > kept around rather than all of them, then the algorithm implementation > should be redesigned so it can make effective use of itertools.tee() As I understand, everything goes into the tee and everything comes out in order, so I don't understand what you mean keeping some value. > In other words, this strikes as a fairly complicated solution to a > rather limited problem. I believe someone else though it was too simple for the stdlib ;-) The solution is a variation of functools.partial. I would certainly look for more concrete use cases, perhaps in the stdlib, before seriously proposing it for inclusion. Terry Jan Reedy From ncoghlan at gmail.com Sun Jun 21 02:53:42 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 21 Jun 2009 10:53:42 +1000 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> <79990c6b0906201516g5554d1damd0cf3c058248fd6e@mail.gmail.com> <4A3D6350.8060004@gmail.com> Message-ID: <4A3D8496.5050809@gmail.com> Terry Reedy wrote: > As I understand, everything goes into the tee and everything comes > out in order, so I don't understand what you mean keeping some value. > itertools.tee() knows where each of the spawned iterators is up to in the sequence so it knows when it can throw away old values. In the degenerate case (such as your cartesian product example) it has to store the whole series in memory but it can usually do better than that (based on how far apart the spawned iterators get). > I believe someone else though it was too simple for the stdlib ;-) > The solution is a variation of functools.partial. Approaching it as a variant of functools.partial certainly sounds like the way to go (and rereading your initial post, I realise you have indeed been doing that all along). It will be interesting to see what your search for concrete use cases turns up. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From greg.ewing at canterbury.ac.nz Sun Jun 21 03:24:38 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Jun 2009 13:24:38 +1200 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <3bdda690906190005v22e7e304t1f318b60c7230d7@mail.gmail.com> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> <20090617011210.GA9427@panix.com> <4A399148.1060504@canterbury.ac.nz> <3bdda690906190005v22e7e304t1f318b60c7230d7@mail.gmail.com> Message-ID: <4A3D8BD6.5010506@canterbury.ac.nz> Carl Johnson wrote: > Actually, the votes have been counted at various times in the past. > Until 2000, the secretary of state for Florida was contracted to count > the votes, but following a Supreme Court ruling, that was > discontinued, and new electronic system from Diebold was put in place. > Then in 2004, there was a lot of grumbling that the Diebold machines > were using Perl internally, so they outsourced the whole vote counting > project to a team in Iran. Unfortunately, the team there hasn't been > answering their emails lately? (And good luck to them, insha'allah!) Latest news is that the Iranian people have demanded a recount, but that turned out to be impossible because the authorities kept the votes in a non-reiterable container. -- Greg From grosser.meister.morti at gmx.net Sun Jun 21 03:51:28 2009 From: grosser.meister.morti at gmx.net (=?UTF-8?B?TWF0aGlhcyBQYW56ZW5iw7Zjaw==?=) Date: Sun, 21 Jun 2009 03:51:28 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <50697b2c0906191405p2bd47b74ga5192cdacca5c570@mail.gmail.com> References: <4A3BD0C4.9090007@gmx.net> <50697b2c0906191229x22c31665o9a60562f1ed7db8f@mail.gmail.com> <4A3BED7C.20002@gmx.net> <50697b2c0906191405p2bd47b74ga5192cdacca5c570@mail.gmail.com> Message-ID: <4A3D9220.8080607@gmx.net> Chris Rebert wrote: > On Fri, Jun 19, 2009 at 12:56 PM, Mathias > Panzenb?ck wrote: >> Chris Rebert wrote: >>> Comprehensions and generator expressions already give us most of the >>> LINQ functionality. Add in `list()` and the ability to `.sort()` lists >>> with a `key` argument and you have the entire thing, except for the >>> one corner case being discussed. Unless I've overlooked something... >> Yes: With LINQ its possible to build a query object out of an LINQ >> expression instead of evaluating it eagerly. This is used primarily to >> generate SQL code while still using syntax native to the host language (C#) >> and preserving type safety (ok the later cannot be done in python). > > One could probably hack that part together with lambdas, the ast > module, and some black magic though. > And are there any use cases besides SQL? Yes: Queries on XML data. So you have the exact same Syntax for queries on simple lists, SQL databases and XML files (but yes, using LINGQ for XML is still much more to write than using something like XPath). I think you can also add your own backends if you like (e.g. for yaml?). And it's all native syntax (no limits on expressiveness and no problems concerning string escaping etc.). -panzi From greg.ewing at canterbury.ac.nz Sun Jun 21 03:56:50 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 21 Jun 2009 13:56:50 +1200 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> Message-ID: <4A3D9362.4080600@canterbury.ac.nz> Terry Reedy wrote: > The issue is more frequent in Py3, which largely shifts from lists to > iterators as the common sequence interchange format. I don't think that's right. As I understand it, things like dict.items() now return *views*, which *are* reiterable (as long as the underlying data doesn't change). -- Greg From steve at pearwood.info Sun Jun 21 05:46:05 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 21 Jun 2009 13:46:05 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906201909.41140.steve@pearwood.info> Message-ID: <200906211346.06205.steve@pearwood.info> On Sun, 21 Jun 2009 04:08:05 am Lie Ryan wrote: > Steven D'Aprano wrote: > >> The advantage of F being callable is that it does not need > >> semantic change, the filtering part will be done before expression > >> just like it is right now. > > > > That's not true. It can't be true. If you want to filter on x**x > > being greater than 100, you need to calculate x**x first. In > > theory, a sufficiently clever compiler could recognise that, say, > > x**x < 100 implies 0 <= x < 3.59728 (approx), but in general, you > > can't predict the value of f(x) without actually calculating f(x). > > Did you read the middle part of the post and the diagram, which > address the question you're asking and how it would be handled? Yes. It made no sense to me at all. Let's make a practical example: c = 1246158315.0 # approximately a week from now L = filter(lambda t: t > c, [time.time() for x in range(20)]) becomes: L = [time.time() as F for x in range(20) if F() > c] How do you expect your proposed syntax to determine whether or not the current time is greater than c without actually checking the current time? Note also that your proposal requires list comps to become like lambda. Using your earlier example: [x**x as F for x in lst if F() < 100] This doesn't bind the value of x**x to the name F, but the expression x**x to F. That makes it like a lambda: lambda x: x**x except the name x is implied and some sort of magic takes place to ensure that by the time you call F(), the appropriate value of x is still around. If this is to apply to generator expressions as well, calling F() could occur some arbitrarily large time later. That means that [time.time() for x in lst] will create a list that looks something like: [1245555766.5681751, 1245555767.2609128, ...] but your proposed: [time.time() as F for x in lst] will create a list something like: [, , ...] -- Steven D'Aprano From stephen at xemacs.org Sun Jun 21 08:10:13 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 21 Jun 2009 15:10:13 +0900 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: <877hz6jewq.fsf@uwakimon.sk.tsukuba.ac.jp> Lie Ryan writes: > The proposal enables comprehension to become filter+map+filter, > map+filter, filter+map, or map-only; eliminating the redundant map in > (map+filter)+(map+filter). The problem with the proposal is that that is *all* it does. I don't see how you plan to disambiguate in cases where the desired operation "really is" (map+filter)+(map+filter). So what you're stuck at is "I want this one operation to be a one-liner." The reply to that is "not every two-line function needs to have special syntax." Unless you can generalize this proposal to handle more general lambdas in a beautiful and Pythonic way, you've already been offered two obvious and nice ways to do it, and another obvious way that is an "ugly" mixed metaphor. All of the other idioms *do* generalize. We just spent several years de-cluttering Python 3, it's too soon to start cluttering it up again, no matter how beautiful it makes this special case. And I happen to think Ben's iterated generator comprehension is the most beautiful of the lot. YMMV of course but so does a that of a lot of other people; getting a consensus for new syntax is going to be impossible if you don't generalize the proposal. From lie.1296 at gmail.com Sun Jun 21 08:25:01 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 16:25:01 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <200906211346.06205.steve@pearwood.info> References: <200906201909.41140.steve@pearwood.info> <200906211346.06205.steve@pearwood.info> Message-ID: Steven D'Aprano wrote: > On Sun, 21 Jun 2009 04:08:05 am Lie Ryan wrote: >> Steven D'Aprano wrote: >>>> The advantage of F being callable is that it does not need >>>> semantic change, the filtering part will be done before expression >>>> just like it is right now. >>> That's not true. It can't be true. If you want to filter on x**x >>> being greater than 100, you need to calculate x**x first. In >>> theory, a sufficiently clever compiler could recognise that, say, >>> x**x < 100 implies 0 <= x < 3.59728 (approx), but in general, you >>> can't predict the value of f(x) without actually calculating f(x). >> Did you read the middle part of the post and the diagram, which >> address the question you're asking and how it would be handled? > > Yes. It made no sense to me at all. > > Let's make a practical example: > > > c = 1246158315.0 # approximately a week from now > L = filter(lambda t: t > c, [time.time() for x in range(20)]) > > becomes: > > L = [time.time() as F for x in range(20) if F() > c] > > How do you expect your proposed syntax to determine whether or not the > current time is greater than c without actually checking the current > time? Of course it will call F(). F() will evaluate time.time() and cache it so future call to F() do not need to call time.time() twice (resulting in different time used for filter and expression). > Note also that your proposal requires list comps to become like lambda. > Using your earlier example: > > [x**x as F for x in lst if F() < 100] > > This doesn't bind the value of x**x to the name F, but the expression > x**x to F. That makes it like a lambda: > > lambda x: x**x > > except the name x is implied and some sort of magic takes place to > ensure that by the time you call F(), the appropriate value of x is > still around. If this is to apply to generator expressions as well, > calling F() could occur some arbitrarily large time later. F() could be called when the filtering takes place or not be called at all, when it does gets called, the expression would be evaluated. In effect this moves forward the expression evaluation to the middle of filtering process. > That means that > > [time.time() for x in lst] > > will create a list that looks something like: > > [1245555766.5681751, 1245555767.2609128, ...] > > > but your proposed: > > [time.time() as F for x in lst] > > will create a list something like: > > [, , ...] > Where did you get that idea? The comprehension would call F before appending it to the list. The equivalent for-loop syntax also clearly stated that: result = [] for x in lst: # F() ensures f(x) will be only ever be called once def F(): nonlocal _cache if not _cache: _cache = f(x) return _cache _cache = None if g(F()): # here F is called before it's appended, # result is the same as calling f(x) # but prevents evaluating f(x) twice result.append(F()) From lie.1296 at gmail.com Sun Jun 21 08:32:35 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 16:32:35 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <877hz6jewq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> <877hz6jewq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Stephen J. Turnbull wrote: > Lie Ryan writes: > > > The proposal enables comprehension to become filter+map+filter, > > map+filter, filter+map, or map-only; eliminating the redundant map in > > (map+filter)+(map+filter). > > The problem with the proposal is that that is *all* it does. I don't > see how you plan to disambiguate in cases where the desired operation > "really is" (map+filter)+(map+filter). So what you're stuck at is "I > want this one operation to be a one-liner." The reply to that is "not > every two-line function needs to have special syntax." The regular map+filter is still available by not using the `as` keyword (or by not calling F). Thus when what you really wanted is two maps, you need to use two comprehensions. The number of maps should == The number of comprehension; currently it is not always possible to do so, when the assumed map and filter ordering doesn't match our required ordering. From lie.1296 at gmail.com Sun Jun 21 08:36:27 2009 From: lie.1296 at gmail.com (Lie Ryan) Date: Sun, 21 Jun 2009 16:36:27 +1000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <87d48yzdeq.fsf@benfinney.id.au> References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> <87d48yzdeq.fsf@benfinney.id.au> Message-ID: Ben Finney wrote: > Lie Ryan writes: > >> Steven D'Aprano wrote: >> >>> The only case it doesn't cover is where the second filter depends on >>> the value of x, and even that can be covered with a *tiny* bit more >>> work: >>> >>> gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >>> result = [y[1] for y in gen if -y[0] < y[1] < y[0]] >>> >>> It requires no new syntax, no changes to the behaviour of list >>> comps, no new meaning on "as", it's understandable and readable. > > I think it would be more readable without index references, but instead > using tuple unpacking:: > > gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) > result = [b for (a, b) in gen if -a < b < a] > > It can even be done as a single expression without (IMO) significantly > affecting readability:: > > result = [ > b for (a, b) in > ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) > if -a < b < a] IMHO, when a comprehension requires more than a single line, it should turn into explicit loop. From 8mayday at gmail.com Sun Jun 21 10:54:00 2009 From: 8mayday at gmail.com (Andrey Popp) Date: Sun, 21 Jun 2009 12:54:00 +0400 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> <87d48yzdeq.fsf@benfinney.id.au> Message-ID: What about where/let expression? [(y, y) for x in some_list if y < 0 where y = f(x)] On Sun, Jun 21, 2009 at 10:36 AM, Lie Ryan wrote: > Ben Finney wrote: >> Lie Ryan writes: >> >>> Steven D'Aprano wrote: >>> >>>> The only case it doesn't cover is where the second filter depends on >>>> the value of x, and even that can be covered with a *tiny* bit more >>>> work: >>>> >>>> gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >>>> result = [y[1] for y in gen if -y[0] < y[1] < y[0]] >>>> >>>> It requires no new syntax, no changes to the behaviour of list >>>> comps, no new meaning on "as", it's understandable and readable. >> >> I think it would be more readable without index references, but instead >> using tuple unpacking:: >> >> ? ? gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >> ? ? result = [b for (a, b) in gen if -a < b < a] >> >> It can even be done as a single expression without (IMO) significantly >> affecting readability:: >> >> ? ? result = [ >> ? ? ? ? b for (a, b) in >> ? ? ? ? ? ? ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >> ? ? ? ? if -a < b < a] > > IMHO, when a comprehension requires more than a single line, it should > turn into explicit loop. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- ? ?????????, ?????? ????. +7 911 740 24 91 From pyideas at rebertia.com Sun Jun 21 11:06:18 2009 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 21 Jun 2009 02:06:18 -0700 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> <87d48yzdeq.fsf@benfinney.id.au> Message-ID: <50697b2c0906210206r71d4ce25g80e873fb70124b4@mail.gmail.com> > On Sun, Jun 21, 2009 at 10:36 AM, Lie Ryan wrote: >> Ben Finney wrote: >>> Lie Ryan writes: >>> >>>> Steven D'Aprano wrote: >>>> >>>>> The only case it doesn't cover is where the second filter depends on >>>>> the value of x, and even that can be covered with a *tiny* bit more >>>>> work: >>>>> >>>>> gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >>>>> result = [y[1] for y in gen if -y[0] < y[1] < y[0]] >>>>> >>>>> It requires no new syntax, no changes to the behaviour of list >>>>> comps, no new meaning on "as", it's understandable and readable. >>> >>> I think it would be more readable without index references, but instead >>> using tuple unpacking:: >>> >>> ? ? gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >>> ? ? result = [b for (a, b) in gen if -a < b < a] >>> >>> It can even be done as a single expression without (IMO) significantly >>> affecting readability:: >>> >>> ? ? result = [ >>> ? ? ? ? b for (a, b) in >>> ? ? ? ? ? ? ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >>> ? ? ? ? if -a < b < a] >> >> IMHO, when a comprehension requires more than a single line, it should >> turn into explicit loop. >> >> >> On Sun, Jun 21, 2009 at 1:54 AM, Andrey Popp<8mayday at gmail.com> wrote: > What about where/let expression? > [(y, y) for x in some_list if y < 0 where y = f(x)] > -- > ? ?????????, ?????? ????. > +7 911 740 24 91 (A) Please don't top-post. (http://en.wikipedia.org/wiki/Top-post) (B) That has the distinct disadvantage of adding a new keyword. I instead prefer the "as" version of the proposal for this reason. Cheers, Chris -- http://blog.rebertia.com From g.brandl at gmx.net Sun Jun 21 11:46:40 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 21 Jun 2009 11:46:40 +0200 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> Message-ID: Lie Ryan schrieb: >>> I hate it. It mixes map/filter style and comprehension style; and the >>> fact it does so in a single line only makes it worse. Not that it would >>> be any better in two lines. >> >> Taking this further, using only map/filter style like this >> >> filter(pred, map(f, seq)) >> >> takes two steps. Why is it so bad that doing it in a listcomp >> >> (el for el in (f(y) for y in seq) if el > 2) >> >> takes two steps as well? >> >> Georg >> > > Comprehension, by its nature, is map+filter in a single expression. Let's stick with "+" for function composition. Then comprehension is, as you say, ``map + filter`` (map after filter). It is *not* ``filter + map`` (filter after map), which is what would be needed in the case discussed in this thread. In the functional waym you can of course combine map and filter in the order you like. > Nested comprehension is (map+filter)+(map+filter). Yes, and in this case, we use it as ``(id+filter) + (map+id)``, which is the easiest way to build a ``filter + map`` with the ``map + filter`` building block. > Mixing the two styles is ugly since it means you have to think in two > separate (though related) paradigms. I mentally translate map and filter into comprehensions anyway (or maybe rather both translate into the same more abstract thing), so I don't see it as ugly. Georg From g.brandl at gmx.net Sun Jun 21 11:47:46 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 21 Jun 2009 11:47:46 +0200 Subject: [Python-ideas] Possible method of distinguishing between set-literals, dict-literals, and odict-literals In-Reply-To: <4A3D8BD6.5010506@canterbury.ac.nz> References: <3bdda690906160454w588b9e75y2658dfbe67455aeb@mail.gmail.com> <87hbyf3dvm.fsf@benfinney.id.au> <20090617011210.GA9427@panix.com> <4A399148.1060504@canterbury.ac.nz> <3bdda690906190005v22e7e304t1f318b60c7230d7@mail.gmail.com> <4A3D8BD6.5010506@canterbury.ac.nz> Message-ID: Greg Ewing schrieb: > Carl Johnson wrote: > >> Actually, the votes have been counted at various times in the past. >> Until 2000, the secretary of state for Florida was contracted to count >> the votes, but following a Supreme Court ruling, that was >> discontinued, and new electronic system from Diebold was put in place. >> Then in 2004, there was a lot of grumbling that the Diebold machines >> were using Perl internally, so they outsourced the whole vote counting >> project to a team in Iran. Unfortunately, the team there hasn't been >> answering their emails lately? (And good luck to them, insha'allah!) > > Latest news is that the Iranian people have demanded a > recount, but that turned out to be impossible because > the authorities kept the votes in a non-reiterable > container. Of course. The most important thing with fraud is to eliminate evidence. Georg From 8mayday at gmail.com Sun Jun 21 11:38:28 2009 From: 8mayday at gmail.com (8mayday at gmail.com) Date: Sun, 21 Jun 2009 09:38:28 +0000 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: <50697b2c0906210206r71d4ce25g80e873fb70124b4@mail.gmail.com> Message-ID: <000e0cd296ec155c31046cd885e6@google.com> On Jun 21, 2009 1:06pm, Chris Rebert wrote: > > On Sun, Jun 21, 2009 at 10:36 AM, Lie Ryanlie.1296 at gmail.com> wrote: > >> Ben Finney wrote: > >>> Lie Ryan lie.1296 at gmail.com> writes: > >>> > >>>> Steven D'Aprano wrote: > >>>> > >>>>> The only case it doesn't cover is where the second filter depends on > >>>>> the value of x, and even that can be covered with a *tiny* bit more > >>>>> work: > >>>>> > >>>>> gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) > >>>>> result = [y[1] for y in gen if -y[0] > >>>>> > >>>>> It requires no new syntax, no changes to the behaviour of list > >>>>> comps, no new meaning on "as", it's understandable and readable. > >>> > >>> I think it would be more readable without index references, but > instead > >>> using tuple unpacking:: > >>> > >>> gen = ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) > >>> result = [b for (a, b) in gen if -a > >>> > >>> It can even be done as a single expression without (IMO) significantly > >>> affecting readability:: > >>> > >>> result = [ > >>> b for (a, b) in > >>> ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) > >>> if -a > >> > >> IMHO, when a comprehension requires more than a single line, it should > >> turn into explicit loop. > >> > >> > >> > On Sun, Jun 21, 2009 at 1:54 AM, Andrey Popp8mayday at gmail.com> wrote: > > What about where/let expression? > > [(y, y) for x in some_list if y > > -- > > ? ?????????, ?????? ????. > > +7 911 740 24 91 > (A) Please don't top-post. (http://en.wikipedia.org/wiki/Top-post) > (B) That has the distinct disadvantage of adding a new keyword. I > instead prefer the "as" version of the proposal for this reason. > Cheers, > Chris > -- > http://blog.rebertia.com (A) Sorry (B) There is no difference, except "where" is widely used in othe languages. Anyway, I think that "where" like functionality would be useful in list comprehensions, lambdas... -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Jun 21 12:51:28 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 21 Jun 2009 19:51:28 +0900 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> <877hz6jewq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <8763epkggf.fsf@uwakimon.sk.tsukuba.ac.jp> Lie Ryan writes: > The regular map+filter is still available by not using the `as` keyword > (or by not calling F). Thus when what you really wanted is two maps, you > need to use two comprehensions. The number of maps should == The number > of comprehension; currently it is not always possible to do so, when the > assumed map and filter ordering doesn't match our required ordering. I understand that; the first part is a trivial consequence of backward compatibility, and the second you've been at pains to explain already. I'm not trying to say you're *wrong*. The appeal of the proposed syntax in the particular case is obvious. Admittedly, I personally don't find it particularly useful. I don't have any problem at all with decomposing what you consider to be a single comprehension into a pipeline of generators. It's efficient and elegant, and it's not clear to me that your construct can generate better byte code than Ben's nested comprehensions. If not, your claim that "this is conceptually a single comprehension, why break it into two or more" seems to me to be founded on quicksand. But that's beside the point, I don't have to like all the constructs that other people find useful. Even Guido has allowed things into Python that he personally dislikes quite a bit. I'm saying it's my impression that it will be *insufficient*. It's like if somebody suggested introducing a unary operator "**" to denote "squared". That's just not useful enough, while a *binary* operator "**" for "power" is useful enough to have been added ages ago. In other words, that kind of logic hasn't been able to justify *proliferation* of syntax in the more than 5 years I've been following Python-Dev (and now Python-Ideas). It only works when the new syntax is sufficiently comprehensive to replace "all the old uglies" in some sense. Even if you can't clear the hurdle in one bound, you need to aim at something like "getting rid of *all* lambda/external function definitions in comprehensions". From p.f.moore at gmail.com Sun Jun 21 16:50:17 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 21 Jun 2009 15:50:17 +0100 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> <79990c6b0906201516g5554d1damd0cf3c058248fd6e@mail.gmail.com> <4A3D6350.8060004@gmail.com> Message-ID: <79990c6b0906210750v7f00ff7fraed646a79cd797fa@mail.gmail.com> 2009/6/21 Terry Reedy : > Nick Coghlan wrote: >> 1. I've never seen a function parameter spec'ed as iterable vs >> reiterable. It's always iterable vs sequence. > > The latter could be over-spec'ed because people are used to that being the > choice, or because 'sequence' is being used as a synonym for 'reiterable' > even though it implies more than is needed. OTOH, it could be because "reiterable" isn't a commonly used concept in Python. It's not entirely clear to me why this is, maybe it's because it's not actually that useful, maybe it's because it's hard to define clearly, or maybe there's another reason. But to some extent this whole thread has the same sort of "solution looking for a problem" feel that earlier threads about reiterability have had. FWIW, the difference between iterators and reiterables is similar to that between C++ input iterators and forward iterators. So it may be worth looking to that in the quest for use cases. However, itertools.tee may cover a number of cases that would need a forward iterator in C++. Paul. From p.f.moore at gmail.com Sun Jun 21 16:59:51 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 21 Jun 2009 15:59:51 +0100 Subject: [Python-ideas] Accessing the result of comprehension's expression from the conditional In-Reply-To: References: <200906200035.51009.steve@pearwood.info> <200906201032.52123.steve@pearwood.info> <87d48yzdeq.fsf@benfinney.id.au> Message-ID: <79990c6b0906210759j21d181adt29459e62c3e79879@mail.gmail.com> 2009/6/21 Lie Ryan : >> It can even be done as a single expression without (IMO) significantly >> affecting readability:: >> >> ? ? result = [ >> ? ? ? ? b for (a, b) in >> ? ? ? ? ? ? ((x, 3*x**2-5*x+4) for x in seq if x % 3 != 2) >> ? ? ? ? if -a < b < a] > > IMHO, when a comprehension requires more than a single line, it should > turn into explicit loop. Hang on - isn't the whole point of this thread that, in your opinion, when a comprehension gets complex, it requires new syntax? :-) Paul. From tjreedy at udel.edu Sun Jun 21 22:12:01 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 21 Jun 2009 16:12:01 -0400 Subject: [Python-ideas] Aid reiteration with new class: gfic In-Reply-To: <79990c6b0906210750v7f00ff7fraed646a79cd797fa@mail.gmail.com> References: <79990c6b0906191337n714004d8y8e8b0f5f754e306e@mail.gmail.com> <79990c6b0906201516g5554d1damd0cf3c058248fd6e@mail.gmail.com> <4A3D6350.8060004@gmail.com> <79990c6b0906210750v7f00ff7fraed646a79cd797fa@mail.gmail.com> Message-ID: Paul Moore wrote: > 2009/6/21 Terry Reedy : >> Nick Coghlan wrote: >>> 1. I've never seen a function parameter spec'ed as iterable vs >>> reiterable. It's always iterable vs sequence. >> The latter could be over-spec'ed because people are used to that being the >> choice, or because 'sequence' is being used as a synonym for 'reiterable' >> even though it implies more than is needed. > > OTOH, it could be because "reiterable" isn't a commonly used concept > in Python. It's not entirely clear to me why this is, maybe it's > because it's not actually that useful, maybe it's because it's hard to > define clearly, or maybe there's another reason. My thought is this. iterable and iterator are separate and complementary concepts: a collection (concrete or virtual) that can be iterated over and the thing that does the iterating. For (great) programming convenience, Python iterators were made into a subcategory of an expanded 'iterable' category. In informal discourse, people sometimes, perhaps often, use 'iterable' in the narrower conceptual sense rather than the broader Python-implementation sense. Or they sort of mean it in the narrow sense but it does not matter because the statement also applies to iterators once they are made into iterables. To be clear in a Python context, when one wants to say something that does not apply to iterators, one can qualify 'iterable' to 'non-iterator iterable' or, less commonly at present, use the synonym 'reiterable'. I say synonym because any Python iterable object is reiterable as long as the information defining it is not altered, destroyed, or 'forgotten' by the process of iteration. > But to some extent this whole thread has the same sort of "solution > looking for a problem" feel that earlier threads about reiterability > have had. It is a solution to a conceptual and didactic problem that I offered for *possible* practical use. But I am not pushing it. Terry Jan Reedy From 8mayday at gmail.com Mon Jun 22 12:16:05 2009 From: 8mayday at gmail.com (Andrey Popp) Date: Mon, 22 Jun 2009 14:16:05 +0400 Subject: [Python-ideas] Introducing where clauses Message-ID: Hello. Reading discussion on python-ideas about "Accessing the result of comprehension's expression from the conditional", I've came to the idea of where clauses, similar to Haskell's. This solves the problem of recalculating of value multiple times. For example, in the following expression: [(f(x), f(x)) for x in some_iterable if f(x) < 2] value f(y) calculates three times -- It is a problem if function f takes much time to compute its value or has side effects. If we would have where clause, we can rewrite expression above with: [(y, y) for x in some_iterable if y < 2 where y = f(x)] I think it is really useful. We can also expand this idea to lambdas or maybe to introducing arbitrary scoping blocks of code. Other thoughts: - Can we use where clauses in lambda definition to allow some kind of multi-line lamdba's? From ncoghlan at gmail.com Mon Jun 22 12:40:05 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 22 Jun 2009 20:40:05 +1000 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: Message-ID: <4A3F5F85.9090406@gmail.com> Andrey Popp wrote: > value f(y) calculates three times -- It is a problem if function f > takes much time to compute its value or has side effects. If we would > have where clause, we can rewrite expression above with: > > [(y, y) for x in some_iterable if y < 2 where y = f(x)] > > I think it is really useful. We can also expand this idea to lambdas > or maybe to introducing arbitrary scoping blocks of code. Or you could just bite the bullet and write a custom generator: def g(iterable): for x in iterable: y = f(x) if y < 2: yield (y, y) Give it a meaningful name and docstring and it can even be self-documenting. Lambdas, comprehensions and expressions in general all have limits - usually deliberate ones. When one runs up against those limits it is a hint that it is time to switch to using multiple statements (typically factored out into a function that can be substituted for the original inline expression) But then, I'll freely confess to not really understanding the apparently common obsession with wanting to be able to do everything as an expression. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ben+python at benfinney.id.au Mon Jun 22 13:01:07 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 22 Jun 2009 21:01:07 +1000 Subject: [Python-ideas] Introducing where clauses References: Message-ID: <87fxdsy1l8.fsf@benfinney.id.au> Andrey Popp <8mayday at gmail.com> writes: > Reading discussion on python-ideas about "Accessing the result of > comprehension's expression from the conditional", I've came to the > idea of where clauses, similar to Haskell's. > > This solves the problem of recalculating of value multiple times. For > example, in the following expression: > > [(f(x), f(x)) for x in some_iterable if f(x) < 2] New syntax isn't necessary to solve the above stated problem. For example, the following existing syntax is also a solution:: [(y, y) for y in (f(x) for x in some_iterable) if y < 2] For the proposed new syntax to be accepted, it would need to be somehow significantly superior to the existing syntax. Can you demonstrate how it's superior? -- \ ?I took a course in speed waiting. Now I can wait an hour in | `\ only ten minutes.? ?Steven Wright | _o__) | Ben Finney From 8mayday at gmail.com Mon Jun 22 13:08:23 2009 From: 8mayday at gmail.com (Andrey Popp) Date: Mon, 22 Jun 2009 15:08:23 +0400 Subject: [Python-ideas] Introducing where clauses In-Reply-To: <87fxdsy1l8.fsf@benfinney.id.au> References: <87fxdsy1l8.fsf@benfinney.id.au> Message-ID: On Mon, Jun 22, 2009 at 3:01 PM, Ben Finney wrote: > Andrey Popp <8mayday at gmail.com> writes: > >> Reading discussion on python-ideas about ?"Accessing the result of >> comprehension's expression from the conditional", I've came to the >> idea of where clauses, similar to Haskell's. >> >> This solves the problem of recalculating of value multiple times. For >> example, in the following expression: >> >> ? ? [(f(x), f(x)) for x in some_iterable if f(x) < 2] > > New syntax isn't necessary to solve the above stated problem. For > example, the following existing syntax is also a solution:: > > ? ?[(y, y) for y in (f(x) for x in some_iterable) if y < 2] > > For the proposed new syntax to be accepted, it would need to be somehow > significantly superior to the existing syntax. Can you demonstrate how > it's superior? Your statement: [(y, y) for y in (f(x) for x in some_iterable) if y < 2] means producing a list by iterating over the generator, which iterates over the some_iterable, it is correct in algorithmic way, but not in semantic. I did not made statement about impossibility of making this kind of things in python right now, without where-clauses. It is only syntactic sugar, but very expressive, I think. From steve at pearwood.info Mon Jun 22 13:34:45 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 22 Jun 2009 21:34:45 +1000 Subject: [Python-ideas] Introducing where clauses In-Reply-To: <4A3F5F85.9090406@gmail.com> References: <4A3F5F85.9090406@gmail.com> Message-ID: <200906222134.45851.steve@pearwood.info> On Mon, 22 Jun 2009 08:40:05 pm Nick Coghlan wrote: > Lambdas, comprehensions and expressions in general all have limits - > usually deliberate ones. When one runs up against those limits it is > a hint that it is time to switch to using multiple statements > (typically factored out into a function that can be substituted for > the original inline expression) +1 > But then, I'll freely confess to not really understanding the > apparently common obsession with wanting to be able to do everything > as an expression. Some things are conceptually a single operation, and those things are good to write as a single expression. Before we had sorted(), it was uncomfortable to write: L.sort() return L when you wanted a sorted list, because "return a sorted list" is conceptually a single operation, even if sorting is non-trivial. The solution to this was to write a helper function, which was made obsolete when sorted() became a built-in. The OP's suggestion: [(f(x), f(x)) for x in some_iterable if f(x) < 2] is not conceptually a single operation, because producing a list of two-tuples containing some value y repeated but only if y is less than 2 is not conceptually simple. If it were, it would be easy to describe the operation with one or two words, instead of the fourteen it took me. I still believe that the right way to solve this is with a pipeline of simple operations: map(lambda obj: (obj, obj), filter(lambda y: y < 2, map(f, some_iterable))) Re-write with temporary variables, itertools, and generator expressions as preferred. -- Steven D'Aprano From 8mayday at gmail.com Mon Jun 22 14:46:44 2009 From: 8mayday at gmail.com (Andrey Popp) Date: Mon, 22 Jun 2009 16:46:44 +0400 Subject: [Python-ideas] Introducing where clauses In-Reply-To: <200906222134.45851.steve@pearwood.info> References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> Message-ID: I think that producing a list of tuples (that is conceptually image of mapping from some_iterable set) is basic operation. But... ok, now we have three ways to produce list below: [(f(x), f(x)) for x in some_iterable if f(x) < 2] 1) > def g(iterable): > for x in iterable: > y = f(x) > if y < 2: > yield (y, y) 2) > [(y, y) for y in (f(x) for x in some_iterable) if y < 2] 3) > map(lambda obj: (obj, obj), > ? ?filter(lambda y: y < 2, > ? ?map(f, some_iterable))) And none of them does not look as obvious as [(f(x), f(x)) for x in some_iterable if f(x) < 2] , doesn't it? While proposed variant with where-clause [(y, y) for x in some_iterable if y < 2 where y = f(x)] looks more naturally than three suggested variants. I give strong emphasis on that fact, that where-clause is only syntactic sugar, suggested for better readability. From jimjjewett at gmail.com Mon Jun 22 14:59:51 2009 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 22 Jun 2009 08:59:51 -0400 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> Message-ID: On Mon, Jun 22, 2009 at 8:46 AM, Andrey Popp<8mayday at gmail.com> wrote: > ? ?[(y, y) for x in some_iterable if y < 2 where y = f(x)] > > looks more naturally than three suggested variants. Only if you are already thinking about SQL. An "as" would be almost as good as a "where", and would better match the way python has evolved so far. But that still doesn't answer the real question. In isolation, I would see either as a real (if perhaps small and isolated) improvement to readability. The catch is that the improvement must be dramatic enough to justify the cost -- which is extra complexity to the language as a whole. That isn't a cost you see as easily, because you're thinking about comprehensions; it is instead a small tax paid by people doing regular function calls and if branching and for loops. It isn't a big tax, but it is cumulative, and python has worked hard to minimize it. -jJ From 8mayday at gmail.com Mon Jun 22 15:16:35 2009 From: 8mayday at gmail.com (Andrey Popp) Date: Mon, 22 Jun 2009 17:16:35 +0400 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> Message-ID: On Mon, Jun 22, 2009 at 4:59 PM, Jim Jewett wrote: > On Mon, Jun 22, 2009 at 8:46 AM, Andrey Popp<8mayday at gmail.com> wrote: >> ? ?[(y, y) for x in some_iterable if y < 2 where y = f(x)] >> >> looks more naturally than three suggested variants. > > Only if you are already thinking about SQL. > > An "as" would be almost as good as a "where", and would better match > the way python has evolved so far. I thinking mostly about Haskell. "as" or "where"- for me, there is no difference. > But that still doesn't answer the real question. ?In isolation, I > would see either as a real (if perhaps small and isolated) improvement > to readability. > > The catch is that the improvement must be dramatic enough to justify > the cost -- which is extra complexity to the language as a whole. > That isn't a cost you see as easily, because you're thinking about > comprehensions; it is instead a small tax paid by people doing regular > function calls and if branching and for loops. ?It isn't a big tax, > but it is cumulative, and python has worked hard to minimize it. I am not about list comprehension only, there are other cases for where-clause, for example lambdas: f = lambda x: (x, y) if x > 0 else (x, 0) where y = g(x) Or maybe more. List comprehension are only small example, that is show the one of use-cases for where-clause. From grosser.meister.morti at gmx.net Mon Jun 22 15:36:25 2009 From: grosser.meister.morti at gmx.net (=?UTF-8?B?TWF0aGlhcyBQYW56ZW5iw7Zjaw==?=) Date: Mon, 22 Jun 2009 15:36:25 +0200 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> Message-ID: <4A3F88D9.6080202@gmx.net> Maybe "let" would be less ambiguous (because what "where" means in SQL). Also let is used in JavaScript (I consider JavaScript very similar to Python and Ruby). -panzi From stephen at xemacs.org Mon Jun 22 16:20:11 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 22 Jun 2009 23:20:11 +0900 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: <87fxdsy1l8.fsf@benfinney.id.au> Message-ID: <87ljnks63o.fsf@uwakimon.sk.tsukuba.ac.jp> Andrey Popp writes: > Your statement: > > [(y, y) for y in (f(x) for x in some_iterable) if y < 2] > > means producing a list by iterating over the generator, which iterates > over the some_iterable, it is correct in algorithmic way, but not in > semantic. That's only true if you think of "iterable" as a generalized (ie, possibly infinite) sequence. However, you can also think of this, not as "iterating over a sequence of values created by iterating over another sequence of values," but rather "iterating 'application' over a sequence of filters." Then the "for" clause functions as a "where". The first "for" in Ben's expression is interpreted as "where y = f(x)", and the second "for" is interpreted as "where x = next(some_iterable)". This is not fully general; it only works at all in a comprehension context, and I'm not entirely sure it works perfectly here. But it troubles me that we already have a way to say "where" at the upper levels of nesting, and you want to introduce a new "where" that is only useful at upper levels of nesting. Another way to put this is that because these are iterables (streams) rather than sequences (ordered (finite) sets), it doesn't make sense to talk about "double interation." There are multiple levels of iterable here, but in the end there's only one iterative process, and all the levels share the same "iteration". From tjreedy at udel.edu Mon Jun 22 17:24:15 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 22 Jun 2009 11:24:15 -0400 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> Message-ID: Andrey Popp wrote: > I think that producing a list of tuples (that is conceptually image of > mapping from some_iterable set) is basic operation. > But... ok, now we have three ways to produce list below: > > [(f(x), f(x)) for x in some_iterable if f(x) < 2] Taking in isolation, there is no reason to produce a list rather than an iterator. > 1) >> def g(iterable): >> for x in iterable: >> y = f(x) >> if y < 2: >> yield (y, y) This does more than the above because g is reusable both with the same iterable and other iterables. > 2) >> [(y, y) for y in (f(x) for x in some_iterable) if y < 2] Though I would probably write the reusable generator I might write this as ygen = (f(x) for x in some_iterable) # or map(f, some_iterable) if f is an existing function ypairs = ((y, y) for y in ygen if y < 2) There are really two ideas: map f to some_iterable make pairs conditionally. There should be no shame in putting each in a separate statement. Nested generators do not really turn this into 'two' iterations, as iterating with ypairs will run the implied loop in synchrony. > 3) >> map(lambda obj: (obj, obj), >> filter(lambda y: y < 2, >> map(f, some_iterable))) > > And none of them does not look as obvious as > > [(f(x), f(x)) for x in some_iterable if f(x) < 2] > > , doesn't it? While proposed variant with where-clause > > [(y, y) for x in some_iterable if y < 2 where y = f(x)] > > looks more naturally than three suggested variants. 'natural' is in the eye of the beholder > I give strong emphasis on that fact, that where-clause is only > syntactic sugar, suggested for better readability. Too much sugar = stomach ache ;-). Terry Jan Reedy From ben+python at benfinney.id.au Tue Jun 23 01:01:09 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 23 Jun 2009 09:01:09 +1000 Subject: [Python-ideas] Introducing where clauses References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> Message-ID: <877hz3yitm.fsf@benfinney.id.au> Terry Reedy writes: > >> [(y, y) for y in (f(x) for x in some_iterable) if y < 2] > > Though I would probably write the reusable generator > I might write this as > > ygen = (f(x) for x in some_iterable) > # or map(f, some_iterable) if f is an existing function > ypairs = ((y, y) for y in ygen if y < 2) > > There are really two ideas: > map f to some_iterable > make pairs conditionally. > > There should be no shame in putting each in a separate statement. Indeed. My only point with the above example is that, for those who *are* desperate to have this all as a single readable statement, the existing syntax supports it nicely. Since the existing syntax already easily supports every case where the proposed syntax would be used, the burden is on those proposing the new syntax to demonstrate benefits significant enough to outweigh the costs of adding bulk to the language. AFAICT, no convincingly significant benefit has been presented to add syntax for this case. > > I give strong emphasis on that fact, that where-clause is only > > syntactic sugar, suggested for better readability. > > Too much sugar = stomach ache ;-). +1 QOTW -- \ ?Holy knit one purl two, Batman!? ?Robin | `\ | _o__) | Ben Finney From cmjohnson.mailinglist at gmail.com Tue Jun 23 01:15:04 2009 From: cmjohnson.mailinglist at gmail.com (Carl Johnson) Date: Mon, 22 Jun 2009 13:15:04 -1000 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> Message-ID: <3bdda690906221615s565bae72v42402af029b253bf@mail.gmail.com> Andrey Popp wrote: > I am not about list comprehension only, there are other cases for > where-clause, for example lambdas: > > ? ?f = lambda x: (x, y) if x > 0 else (x, 0) where y = g(x) Yuck. This reminds me of why I gave up on the Haskell tutorial I was working through. The reading of this line keeps bouncing back and forth. "OK, function f, passing in x, returning x, y? Wait? What's y? Is that from an external scope? Anyway, here's an if-else clause, and oh, there's the y! It's the same as g(x). OK, so where all did they use y? Hmm, lets see, looks like just the one spot?" This would be much easier to grasp as a function: def f(x): y = g(x) if y > 0: return x, y else: return x, 0 This version makes the parallelism between the two return values much more clear: "Oh, OK, it's always going to return x as the first in the tuple, and the second value will be either g(x) or 0, whichever is greater." We might even re-write it as def f(x): y = g(x) r = y if y > 0 else 0 return x, r to make it shorter. -- Carl From g.brandl at gmx.net Tue Jun 23 09:11:07 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 23 Jun 2009 07:11:07 +0000 Subject: [Python-ideas] Introducing where clauses In-Reply-To: <3bdda690906221615s565bae72v42402af029b253bf@mail.gmail.com> References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> <3bdda690906221615s565bae72v42402af029b253bf@mail.gmail.com> Message-ID: Carl Johnson schrieb: > Andrey Popp wrote: > >> I am not about list comprehension only, there are other cases for >> where-clause, for example lambdas: >> >> f = lambda x: (x, y) if x > 0 else (x, 0) where y = g(x) > > Yuck. This reminds me of why I gave up on the Haskell tutorial I was > working through. The reading of this line keeps bouncing back and > forth. "OK, function f, passing in x, returning x, y? Wait? What's y? > Is that from an external scope? Anyway, here's an if-else clause, and > oh, there's the y! It's the same as g(x). OK, so where all did they > use y? Hmm, lets see, looks like just the one spot?" I know what you mean :) I think "where" is best used where you all but know exactly what the where-bound name refers to, but have to spell it out for the stupid computer somewhere... Georg From gerald.britton at gmail.com Tue Jun 23 15:12:56 2009 From: gerald.britton at gmail.com (Gerald Britton) Date: Tue, 23 Jun 2009 09:12:56 -0400 Subject: [Python-ideas] Introducing where clauses In-Reply-To: References: <4A3F5F85.9090406@gmail.com> <200906222134.45851.steve@pearwood.info> <3bdda690906221615s565bae72v42402af029b253bf@mail.gmail.com> Message-ID: <5d1a32000906230612g38c8260ao44a28ce24b85bca3@mail.gmail.com> Personally I like the idea of the "where" clause. It works well in Haskell since it is tied closely to how functions are often defined in mathematics. e.g. Area of a circle = pi*r**2 where pi is 3.14159.... and r is the radius of the circle In Haskell, it makes for concise function definitions. IIRC defining functions without the "where" clause in Haskell is a Hassle with a capital "H". However Python suffers from no such problem. Though I like the idea as a concept, I see it as syntactic sugar for Python that is essentially a solution in search of a problem. On Tue, Jun 23, 2009 at 3:11 AM, Georg Brandl wrote: > Carl Johnson schrieb: >> Andrey Popp wrote: >> >>> I am not about list comprehension only, there are other cases for >>> where-clause, for example lambdas: >>> >>> ? ?f = lambda x: (x, y) if x > 0 else (x, 0) where y = g(x) >> >> Yuck. This reminds me of why I gave up on the Haskell tutorial I was >> working through. The reading of this line keeps bouncing back and >> forth. "OK, function f, passing in x, returning x, y? Wait? What's y? >> Is that from an external scope? Anyway, here's an if-else clause, and >> oh, there's the y! It's the same as g(x). OK, so where all did they >> use y? Hmm, lets see, looks like just the one spot?" > > I know what you mean :) ?I think "where" is best used where you all > but know exactly what the where-bound name refers to, but have to spell > it out for the stupid computer somewhere... > > Georg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Gerald Britton From kristjan at ccpgames.com Tue Jun 23 22:23:31 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 23 Jun 2009 20:23:31 +0000 Subject: [Python-ideas] add a list.swap() method Message-ID: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> Hello there. Please see my feature request: http://bugs.python.org/issue6326 The idea is to speed up the swapping of list elemenst so that a.swap(b) is equivalent to a[:], b[:] = b[:], a[:] but without all the overhead of creating slices, assigning them and so forth. In particular, it can help reduce the cost of creating instances of list subclasses, from raw lists: class mylist(list): def first(self): return self[0] m = mylist(source_list) This certainly creates m, but does so by copying source_list. If all we want to do is turn source_list into a mylist instance, it is much quicker to: m = mylist() m.swap(source_list) See the above issue for initial comments, especially concerns on how this can bypass all kind of insertion semantics of list subclasses. Cheers, Kristj?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at rcn.com Wed Jun 24 00:02:18 2009 From: python at rcn.com (Raymond Hettinger) Date: Tue, 23 Jun 2009 15:02:18 -0700 Subject: [Python-ideas] add a list.swap() method References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> Message-ID: [Kristj?n Valur J?nsson] > The idea is to speed up the swapping of list elemenst so that > a.swap(b) > is equivalent to > a[:], b[:] = b[:], a[:] > but without all the overhead of creating slices, assigning them and so forth. The technique is useful and very fast. I used something similar in setobject.c in the set_swap_bodies() function, but it was sufficiently dangerous to subclass invariants that it was not exposed to the end-user. Subclasses written as C extensions are the most in danger because a swap could break their internal invariants and possibly crash the code. Pure python list subclasses are only in danger of breaking without crashing. While Python is a consenting adults language, I think the basic objects like lists should be kept free of dangerous or hard-to-use constructs. The problem with the OP's example is that it only makes sense in interactions between a list and a list subclass that won't be broken by it. For straight list-to-list interactions, it is better to write "a,b = b,a" (though this is not exactly the same thing since the identity of a and b will change, not just their contents). For list subclass uses (a more advanced topic), some example will work and some won't (I gave two failing examples in the tracker discussion). When the list subclass is a C extension written by a third-party, it may not be possible to know whether or not a swap will break invariants. That's a killer. There are a number of places in the language where swapping could be used (frozenset to set conversions for example) but the technique is fragile and probably should not become part of the language as distributed. In the OP's use case, it is not hard to build a C extension for his subclasses and include a swap() method there. That is a much less fragile design because the subclass already knows about its own internal invariants and it can verify that its input source is an exact list. An OOP design principle is that a method should be added to the class that has the most knowledge about all of the inputs (in this case, the C extension subclass knows more than general purpose lists). A side benefit of this design (putting swap() in the list subclass instead of standard list objects) is that this avoids putting dangerous optimizations in the hands of beginning users (i.e. it keeps the list API simple and clean). Raymond From aahz at pythoncraft.com Wed Jun 24 01:15:21 2009 From: aahz at pythoncraft.com (Aahz) Date: Tue, 23 Jun 2009 16:15:21 -0700 Subject: [Python-ideas] add a list.swap() method In-Reply-To: References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> Message-ID: <20090623231521.GA10148@panix.com> On Tue, Jun 23, 2009, Raymond Hettinger wrote: > > [Kristj?n Valur J?nsson] >> >> The idea is to speed up the swapping of list elemenst so that >> a.swap(b) >> is equivalent to >> a[:], b[:] = b[:], a[:] >> but without all the overhead of creating slices, assigning them and so forth. > > The problem with the OP's example is that it only makes sense in > interactions between a list and a list subclass that won't be broken > by it. For straight list-to-list interactions, it is better to write > "a,b = b,a" (though this is not exactly the same thing since the > identity of a and b will change, not just their contents). For list > subclass uses (a more advanced topic), some example will work and some > won't (I gave two failing examples in the tracker discussion). When > the list subclass is a C extension written by a third-party, it may > not be possible to know whether or not a swap will break invariants. > That's a killer. Thanks for the detailed response and explanation of why Kristjan wants the feature and why it isn't the same thing as the standard tuple swap. Your assessment makes sense, and I agree that this doesn't belong as a standard list feature. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From tjreedy at udel.edu Wed Jun 24 02:09:07 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 23 Jun 2009 20:09:07 -0400 Subject: [Python-ideas] add a list.swap() method In-Reply-To: References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> Message-ID: Raymond Hettinger wrote: > > [Kristj?n Valur J?nsson] >> The idea is to speed up the swapping of list elemenst so that >> a.swap(b) >> is equivalent to >> a[:], b[:] = b[:], a[:] >> but without all the overhead of creating slices, assigning them and so >> forth. > > The technique is useful and very fast. I used something similar in > setobject.c > in the set_swap_bodies() function, but it was sufficiently dangerous to > subclass invariants that it was not exposed to the end-user. > > Subclasses written as C extensions are the most in danger because a > swap could break their internal invariants and possibly crash the code. > Pure python list subclasses are only in danger of breaking without > crashing. > > While Python is a consenting adults language, I think the basic objects > like lists should be kept free of dangerous or hard-to-use constructs. > > The problem with the OP's example is that it only makes sense in > interactions > between a list and a list subclass that won't be broken by it. For > straight > list-to-list interactions, it is better to write "a,b = b,a" (though > this is > not exactly the same thing since the identity of a and b will change, > not just their contents). For list subclass uses (a more advanced topic), > some example will work and some won't (I gave two failing examples in the > tracker discussion). When the list subclass is a C extension written by a > third-party, it may not be possible to know whether or not a swap > will break invariants. That's a killer. > > There are a number of places in the language where swapping could > be used (frozenset to set conversions for example) but the technique > is fragile and probably should not become part of the language as > distributed. > > In the OP's use case, it is not hard to build a C extension for his > subclasses > and include a swap() method there. That is a much less fragile design > because > the subclass already knows about its own internal invariants and it can > verify that its input source is an exact list. An OOP design principle > is that a > method should be added to the class that has the most knowledge about > all of the inputs (in this case, the C extension subclass knows more than > general purpose lists). A side benefit of this design (putting swap() > in the > list subclass instead of standard list objects) is that this avoids > putting dangerous optimizations in the hands of beginning users (i.e. it > keeps the list API simple and clean). I learned from this. This seems like an appropriate wiki topic or recipe for advanced extension writers. From steve at pearwood.info Wed Jun 24 02:19:45 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 24 Jun 2009 10:19:45 +1000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> Message-ID: <200906241019.46148.steve@pearwood.info> On Wed, 24 Jun 2009 06:23:31 am Kristj?n Valur J?nsson wrote: > Hello there. > Please see my feature request: http://bugs.python.org/issue6326 > > The idea is to speed up the swapping of list elemenst so that > a.swap(b) > is equivalent to > a[:], b[:] = b[:], a[:] > but without all the overhead of creating slices, assigning them and > so forth. In particular, it can help reduce the cost of creating > instances of list subclasses, from raw lists: > > class mylist(list): > def first(self): > return self[0] > > m = mylist(source_list) Is that overhead really significant enough that it needs a special operation to avoid it? In the tracker, you claim a significant performance gain, but I'd like to hear more details. > This certainly creates m, but does so by copying source_list. Only the pointers, not the objects pointed to. But I'm sure you already know this :) > If all > we want to do is turn source_list into a mylist instance, it is much > quicker to: m = mylist() > m.swap(source_list) This has a side-effect of turning source_list into an empty mylist. I don't like that one bit. > See the above issue for initial comments, especially concerns on how > this can bypass all kind of insertion semantics of list subclasses. Can you address those concerns? One problem with swap() being a list method is that it allows people to arbitrarily swap the contents of lists around, breaking subclass invariants. I have an alternative suggestion: put the swap into list.__new__, so it can only be done at list initialisation time. list.__new__(cls, source) currently returns an empty list of type cls, which is then initialised by list.__init__(self, source), presumably by copying source. Perhaps list.__new__ could take an additional argument that says "swap the contents of the new list to use source without copying". That means, instead of writing: a = mylist() a.swap([1, 2, 3]) you would put the fast initialisation logic into the class: class mylist(list): def __new__(cls, source): return list.__new__(cls, source, swap=True) def __init__(self, source): pass # avoid calling list.__init__ Existing code should work unchanged if the default for swap is False. -- Steven D'Aprano From gagsl-py2 at yahoo.com.ar Wed Jun 24 17:15:14 2009 From: gagsl-py2 at yahoo.com.ar (gagsl-py2 at yahoo.com.ar) Date: Wed, 24 Jun 2009 08:15:14 -0700 (PDT) Subject: [Python-ideas] enhance filecmp to support text-and-universal-newline-mode file comparison Message-ID: <917637.95957.qm@web32808.mail.mud.yahoo.com> [Please keep the discussion in the list. Also, please avoid top posting (corrected below)] > On 6/20/09, Gabriel Genellina > wrote: > > En Thu, 18 Jun 2009 11:04:34 -0300, zhong nanhai > > > escribi?: > > > >> So is it a good idea to enhance the filecmp to > support > >> universal-newline-mode?If so, we can compare > different files from > >> different operation systems and if they have the > same content, the > >> filecmp.cmp would return true. > > > > With aid from itertools.izip_longest, it's a one-line > recipe: > > > > py> print repr(open("one.txt","rb").read()) > > 'hello\nworld!\nlast line\n' > > py> print repr(open("two.txt","rb").read()) > > 'hello\r\nworld!\r\nlast line\r\n' > > py> import filecmp > > py> filecmp.cmp("one.txt", "two.txt", False) > > False > > py> from itertools import izip_longest > > py> f1 = open("one.txt", "rU") > > py> f2 = open("two.txt", "rU") > > py> > > py> print all(line1==line2 for line1,line2 in > izip_longest(f1,f2)) > > True > > > > Currently filecmp considers both files as binary, not > text; if they differ > > in size they're considered different and the contents > are not even read. > > > > If you want a generic text-mode file comparison, there > are other factors > > to consider in addition to line endings: character > encoding, BOM, > > character case, whitespace... All of those may be > considered "irrelevant > > differences" by some people. A generic text file > comparison should take > > all of them into account. --- El vie 19-jun-09, zhong nanhai escribi?: > Thanks for you suggestion. > You are right and there are a lot of things to consider if > we want to > make filecmp support text comparision.But I think we can > just do some > little feature enhancement,e.g. only? the > universal-newline mode. I am > not clear the way filecmp implement the file comparision. > So, you can > tell me more about that. > And if in the source of filecmp, it compare files just by > reading them > line by line, then we can do some further comparisons when > encountering newline flag(means the end of a line). You can see it yourself, in lib/filecmp.py in your Python installation. It does a binary comparison only -- and it does not read anything if file sizes differ. A text comparison should use a different algorithm; the code above already ignores end-of-line differences and breaks as soon as two lines differ. One could enhance it to add support for other options as menctioned earlier. -- Gabriel Genellina ____________________________________________________________________________________ ?Viv? la mejor experiencia en la web! Descarg? gratis el nuevo Internet Explorer 8 http://downloads.yahoo.com/ieak8/?l=ar From kristjan at ccpgames.com Wed Jun 24 18:26:55 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 24 Jun 2009 16:26:55 +0000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <20090623231521.GA10148@panix.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <20090623231521.GA10148@panix.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F057FB5FB0F@exchis.ccp.ad.local> I agree that as presented, it is a bit radical due to the concerns raised by Raymond and others. But I still think it is a useful recipe for those that know what they are doing, and then perhaps as the standalone function, SwapLists(a,b) like we implemented in-house for our purposes. I wonder, where do such recipes belong? K > -----Original Message----- > From: python-ideas-bounces+kristjan=ccpgames.com at python.org > [mailto:python-ideas-bounces+kristjan=ccpgames.com at python.org] On > Behalf Of Aahz > Sent: 23. j?n? 2009 23:15 > To: python-ideas at python.org > Subject: Re: [Python-ideas] add a list.swap() method > > On Tue, Jun 23, 2009, Raymond Hettinger wrote: > > > > [Kristj?n Valur J?nsson] > >> > >> The idea is to speed up the swapping of list elemenst so that > >> a.swap(b) > >> is equivalent to > >> a[:], b[:] = b[:], a[:] > >> but without all the overhead of creating slices, assigning them and > so forth. > > > > The problem with the OP's example is that it only makes sense in > > interactions between a list and a list subclass that won't be broken > > by it. For straight list-to-list interactions, it is better to write > > "a,b = b,a" (though this is not exactly the same thing since the > > identity of a and b will change, not just their contents). For list > > subclass uses (a more advanced topic), some example will work and > some > > won't (I gave two failing examples in the tracker discussion). When > > the list subclass is a C extension written by a third-party, it may > > not be possible to know whether or not a swap will break invariants. > > That's a killer. > > Thanks for the detailed response and explanation of why Kristjan wants > the feature and why it isn't the same thing as the standard tuple swap. > Your assessment makes sense, and I agree that this doesn't belong as a > standard list feature. From aahz at pythoncraft.com Wed Jun 24 18:50:20 2009 From: aahz at pythoncraft.com (Aahz) Date: Wed, 24 Jun 2009 09:50:20 -0700 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5FB0F@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <20090623231521.GA10148@panix.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FB0F@exchis.ccp.ad.local> Message-ID: <20090624165019.GA13980@panix.com> On Wed, Jun 24, 2009, Kristj?n Valur J?nsson wrote: > > I agree that as presented, it is a bit radical due to the concerns > raised by Raymond and others. But I still think it is a useful recipe > for those that know what they are doing, and then perhaps as the > standalone function, SwapLists(a,b) like we implemented in-house for > our purposes. I wonder, where do such recipes belong? Python Cookbook, for starters: http://www.activestate.com/ASPN/Python/Cookbook/ You might also upload a small module to PyPI. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From wuwei23 at gmail.com Thu Jun 25 09:08:18 2009 From: wuwei23 at gmail.com (alex23) Date: Thu, 25 Jun 2009 00:08:18 -0700 (PDT) Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> Message-ID: <934532c4-a208-444b-915d-9c730c70a59b@d2g2000pra.googlegroups.com> Kristj?n Valur J?nsson wrote: > The idea is to speed up the swapping of list elemenst so that > a.swap(b) > is equivalent to > a[:], b[:] = b[:], a[:] > but without all the overhead of creating slices, assigning them and so forth. I think I'd rather use a mapping to hold the lists so I could do a straight re-assignment: d['a'], d['b'] = d['b'], d['a'] Of course, all references to the lists would have to be brokered through the mapping object, but I tend to find it easier to do so anyway. From wuwei23 at gmail.com Thu Jun 25 09:00:58 2009 From: wuwei23 at gmail.com (alex23) Date: Thu, 25 Jun 2009 00:00:58 -0700 (PDT) Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> Message-ID: <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> Kristj?n Valur J?nsson wrote: > The idea is to speed up the swapping of list elemenst so that > a.swap(b) > is equivalent to > a[:], b[:] = b[:], a[:] > but without all the overhead of creating slices, assigning them and so forth. I think I'd use a mapping to hold the lists so I could do a straight re-assignment: d['a'], d['b'] = d['b'], d['a'] Of course, all references to the lists would have to be brokered through the mapping object, but I tend to find it easier to do so anyway. From kristjan at ccpgames.com Thu Jun 25 11:05:41 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 25 Jun 2009 09:05:41 +0000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> > -----Original Message----- > From: python-ideas-bounces+kristjan=ccpgames.com at python.org > [mailto:python-ideas-bounces+kristjan=ccpgames.com at python.org] On > Behalf Of alex23 > Sent: 25. j?n? 2009 07:01 > To: python-ideas at python.org > Subject: Re: [Python-ideas] add a list.swap() method > > Kristj?n Valur J?nsson wrote: > > The idea is to speed up the swapping of list elemenst so that > > a.swap(b) > > is equivalent to > > a[:], b[:] = b[:], a[:] > > but without all the overhead of creating slices, assigning them and > so forth. > > I think I'd use a mapping to hold the lists so I could do a straight > re-assignment: > > d['a'], d['b'] = d['b'], d['a'] > This is roughly equivalent to a, b = b, a right? That's a completely different thing. Also, the whole point of this excersise is performance. This is why in our case, for example, we were using a list subclass (CRowset) in stead of wrapping the list, to shave off cpu cycles when accessing rows and columns in a rowset. A CRowset is essentially a list, with an extra attribute (header). So, maybe I should rephrase the "idea." The idea is for example to speed up the initialization of lists (and subclasses of lists) from other lists when the source list's contents is to be discarded. The semantics of a.swap(b) are the same as a[:], b[:] = b[:], a[:] but it a) is orders of magnitude faster b) skips any special methods (such as __setslice__) and so may break class invariants of list subclasses, as pointed out on this list. K From ncoghlan at gmail.com Thu Jun 25 12:32:55 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Jun 2009 20:32:55 +1000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> Message-ID: <4A435257.2070102@gmail.com> Kristj?n Valur J?nsson wrote: > So, maybe I should rephrase the "idea." > The idea is for example to speed up the initialization of lists (and subclasses of lists) from other lists when the source list's contents is to be discarded. The semantics of > a.swap(b) > are the same as > a[:], b[:] = b[:], a[:] > but it > a) is orders of magnitude faster > b) skips any special methods (such as __setslice__) and so may break class invariants of list subclasses, as pointed out on this list. As Raymond posted, the potential breakage is a major downside (and almost certainly a deal breaker without some safety rails, and possibly still a deal breaker even with them). The name also concerns me, since I think it is misleading as to the intended usage of the method. My suggestions: 1. Make it a class method "from_list". Document its semantics as something like: @classmethod def from_list(cls, other): """Fast but destructive conversion of a list instance to a different type of list instance""" self = cls() self[:] = other[:] other[:] = [] return self 2. Put in some safety rails, similar to what we do to prevent hashing of classes that override __eq__ without overriding __hash__. If __setitem__ and __setslice__ on cls or other aren't the same as the base class definitions, don't allow the operation to proceed (i.e. throw a TypeError saying that the operation needs list instances that use the standard value setting semantics). 3. Possibly provide some C API level tools to make it easier for a list subclass to write their own fast constructor based on this idea I'm not sure it is actually practical to make munging about with list internals like this "safe" enough to be comfortably exposed at the Python level, but ideas like the above would at least be a step in that direction. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From aahz at pythoncraft.com Thu Jun 25 15:47:00 2009 From: aahz at pythoncraft.com (Aahz) Date: Thu, 25 Jun 2009 06:47:00 -0700 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> Message-ID: <20090625134700.GA17910@panix.com> On Thu, Jun 25, 2009, Kristj?n Valur J?nsson wrote: > From: alex23 >> Kristj?n Valur J?nsson wrote: >>> >>> The idea is to speed up the swapping of list elemenst so that >>> a.swap(b) >>> is equivalent to >>> a[:], b[:] = b[:], a[:] >>> but without all the overhead of creating slices, assigning them and >>> so forth. >> >> I think I'd use a mapping to hold the lists so I could do a straight >> re-assignment: >> >> d['a'], d['b'] = d['b'], d['a'] > > This is roughly equivalent to > a, b = b, a > right? Yes and no. The point is that anyone holding a reference to ``d`` will see the changes to d['a'] and d['b'] (this would also apply to using an instance to proxy the changes through attributes). It's the usual trick for handling wholesale updates to mutable objects. Therefore it doesn't have the problem of the plain tuple-swap of losing the references. For any use-case that really is swapping as opposed to the fast initialization that you want, alex23's idea is a good kludge, which limits the scope of usefulness for what you want. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From rrr at ronadam.com Thu Jun 25 17:11:03 2009 From: rrr at ronadam.com (Ron Adam) Date: Thu, 25 Jun 2009 10:11:03 -0500 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <4A435257.2070102@gmail.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> <4A435257.2070102@gmail.com> Message-ID: <4A439387.80006@ronadam.com> Nick Coghlan wrote: > Kristj?n Valur J?nsson wrote: >> So, maybe I should rephrase the "idea." >> The idea is for example to speed up the initialization of lists (and subclasses of lists) from other lists when the source list's contents is to be discarded. The semantics of >> a.swap(b) >> are the same as >> a[:], b[:] = b[:], a[:] >> but it >> a) is orders of magnitude faster >> b) skips any special methods (such as __setslice__) and so may break class invariants of list subclasses, as pointed out on this list. > I'm not sure it is actually practical to make munging about with list > internals like this "safe" enough to be comfortably exposed at the > Python level, but ideas like the above would at least be a step in that > direction. It sounds to me like what is really desired in this case is not a copy/swap operation but a type mutate operation on the data. So instead of copying the data, the data's type is mutated to another type providing that the data format is compatible. So the above becomes something like... a_type = type(a) b_type = type(b) a.mutate_type_to(b_type) b.mutate_type_to(a_type) But the more common use would be just to convert a single data object to another similar data object in a much more efficient way. Would that even be possible? A class would need to know what other classes it is data compatible with somehow. Possibly that could be done by registering that before use. Then it would need the machinery to do that, which I have no idea about. The advantage is it's a more general way to approach the problem that could save a lot of data copying in more situations. Ok, it's a wild idea of sorts, but this is the idea list. ;-) Cheers, Ron From tjreedy at udel.edu Thu Jun 25 19:54:02 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jun 2009 13:54:02 -0400 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <4A439387.80006@ronadam.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> <4A435257.2070102@gmail.com> <4A439387.80006@ronadam.com> Message-ID: Ron Adam wrote: > > > Nick Coghlan wrote: >> Kristj?n Valur J?nsson wrote: >>> So, maybe I should rephrase the "idea." >>> The idea is for example to speed up the initialization of lists (and >>> subclasses of lists) from other lists when the source list's contents >>> is to be discarded. The semantics of >>> a.swap(b) >>> are the same as >>> a[:], b[:] = b[:], a[:] >>> but it a) is orders of magnitude faster >>> b) skips any special methods (such as __setslice__) and so may break >>> class invariants of list subclasses, as pointed out on this list. > It sounds to me like what is really desired in this case is not a > copy/swap operation but a type mutate operation on the data. So instead > of copying the data, the data's type is mutated to another type > providing that the data format is compatible. 'Compatible' is the keyword. As I understand him, Kristj?n's problem is initializating list subclasses with additional fields, which are *not* compatible. Class mutation is already possible for user-defined classes. From tjreedy at udel.edu Thu Jun 25 19:57:30 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jun 2009 13:57:30 -0400 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> Message-ID: Kristj?n Valur J?nsson wrote: > >> -----Original Message----- > > The idea is for example to speed up the initialization of lists (and > subclasses of lists) from other lists when the source list's contents > is to be discarded. Which is why 'swap' is confusing. You want to swap list-body pointers, which is meaningless in Python itself. Nick's 'fromlist' is better. It describes the effect without reference to the backstage magic. From tjreedy at udel.edu Thu Jun 25 20:02:53 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jun 2009 14:02:53 -0400 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <4A435257.2070102@gmail.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> <4A435257.2070102@gmail.com> Message-ID: Nick Coghlan wrote: > My suggestions: > > 1. Make it a class method "from_list". Document its semantics as > something like: > > @classmethod > def from_list(cls, other): > """Fast but destructive conversion of a list instance > to a different type of list instance""" > self = cls() > self[:] = other[:] > other[:] = [] > return self > > 2. Put in some safety rails, similar to what we do to prevent hashing of > classes that override __eq__ without overriding __hash__. If __setitem__ > and __setslice__ on cls or other aren't the same as the base class > definitions, don't allow the operation to proceed (i.e. throw a > TypeError saying that the operation needs list instances that use the > standard value setting semantics). Suppose 'fromlist' literally meant from 'from a list object' and not from list subclasses. Would not that be clearly safe? Kristj?n, would that restriction be okay for your purposes, or do you really need from subclass to subclass? > 3. Possibly provide some C API level tools to make it easier for a list > subclass to write their own fast constructor based on this idea > > I'm not sure it is actually practical to make munging about with list > internals like this "safe" enough to be comfortably exposed at the > Python level, but ideas like the above would at least be a step in that > direction. > > Cheers, > Nick. > From kristjan at ccpgames.com Thu Jun 25 21:52:25 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 25 Jun 2009 19:52:25 +0000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> Message-ID: <930F189C8A437347B80DF2C156F7EC7F057FB5FDAA@exchis.ccp.ad.local> > -----Original Message----- > From: python-ideas-bounces+kristjan=ccpgames.com at python.org > [mailto:python-ideas-bounces+kristjan=ccpgames.com at python.org] On > Behalf Of Terry Reedy > Sent: 25. j?n? 2009 17:58 > To: python-ideas at python.org > Subject: Re: [Python-ideas] add a list.swap() method > > Kristj?n Valur J?nsson wrote: > > > >> -----Original Message----- > > > > The idea is for example to speed up the initialization of lists (and > > subclasses of lists) from other lists when the source list's contents > > is to be discarded. > > Which is why 'swap' is confusing. You want to swap list-body pointers, > which is meaningless in Python itself. Nick's 'fromlist' is better. It > describes the effect without reference to the backstage magic. It is only one application, the other is a fast implementation of a[:], b[:] = b[:], a[:] There is precedence, I think. list.reverse() is semantically identical to list[:] = reversed(list) but it doesn't invoke any magic methods, and it relies on "backstage magic" to do its work in the most efficient way. Same goes for sort(), really. The difference with swap(), is that it will also magically affect an operand list. But I think that both reverse() and sort() can be dangerous to list subclasses with class invariants. I think there are other cases. list.pop() doesn't appear to invoke any magic methods either, and so must be a killer for list invariants. K From josiah.carlson at gmail.com Thu Jun 25 23:14:34 2009 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Thu, 25 Jun 2009 14:14:34 -0700 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5FDAA@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com> <930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local> <930F189C8A437347B80DF2C156F7EC7F057FB5FDAA@exchis.ccp.ad.local> Message-ID: -1 I've done some pretty shady things with Python over the years, but swapping the contents of two lists (or list subclasses) isn't one. That this can be implemented in a properly formatted 2-line function (modulo your function naming style): def SwapLists(a, b): a[:], b[:] = b[:], a[:] ... necessarily requires me to invoke "not all x line functions should be built-in (or methods)". As for using list.sort(), list.reverse(), or even list.pop() as examples of why list.swap() should exist, that doesn't fly. Sorting, reversing, and popping are *very* common operations in Python code. Swapping list contents? ...not so common. If you're really swapping contents that often, you may want to consider a different design with a better algorithm (swap 2 pointers rather than n+m), rather than speeding up the slow algorithm. - Josiah 2009/6/25 Kristj?n Valur J?nsson : >> -----Original Message----- >> From: python-ideas-bounces+kristjan=ccpgames.com at python.org >> [mailto:python-ideas-bounces+kristjan=ccpgames.com at python.org] On >> Behalf Of Terry Reedy >> Sent: 25. j?n? 2009 17:58 >> To: python-ideas at python.org >> Subject: Re: [Python-ideas] add a list.swap() method >> >> Kristj?n Valur J?nsson wrote: >> > >> >> -----Original Message----- >> > >> > The idea is for example to speed up the initialization of lists (and >> > subclasses of lists) from other lists when the source list's contents >> > is to be discarded. >> >> Which is why 'swap' is confusing. ?You want to swap list-body pointers, >> which is meaningless in Python itself. Nick's 'fromlist' is better. It >> describes the effect without reference to the backstage magic. > > > It is only one application, the other is a fast implementation of > a[:], b[:] = b[:], a[:] > > There is precedence, I think. > list.reverse() is semantically identical to > list[:] = reversed(list) > > but it doesn't invoke any magic methods, and it relies on "backstage magic" to do its work in the most efficient way. > Same goes for sort(), really. > > The difference with swap(), is that it will also magically affect an operand list. ?But I think that both reverse() and sort() can be dangerous to list subclasses with class invariants. > > I think there are other cases. ?list.pop() doesn't appear to invoke any magic methods either, and so must be a killer for list invariants. > > K > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From python at rcn.com Thu Jun 25 23:58:24 2009 From: python at rcn.com (Raymond Hettinger) Date: Thu, 25 Jun 2009 14:58:24 -0700 Subject: [Python-ideas] add a list.swap() method References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local><3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com><930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local><930F189C8A437347B80DF2C156F7EC7F057FB5FDAA@exchis.ccp.ad.local> Message-ID: <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> [Josiah Carlson] > If you're really swapping contents that often, you may want to > consider a different design with a better algorithm (swap 2 pointers > rather than n+m), rather than speeding up the slow algorithm. That is what the OP was requesting. He needs a pointer swap for speed. His problem is that his input is a regular list and the target is a custom list subclass. Python currently provides no way for the subclass to steal the pointer from a regular list. My recommended approach for him is to write a simple, reusable list subclass in C, one that knows how to steal a pointer from a regular list. Then he could subclass from that C extension: class MyFastListBasedThingie(myextensions.SwappableList): def __init__(self, some_regular_list): if type(some_regular_list) == list: # an exact match self.swap_list_pointers(some_regular_list) # the fast way else: self.extend(some_regular_list) # the slow way del some_regular_list[:] The advantage of this design is that it puts responsibility for swapping in the hands of the class that knows all of the relevant information (MyFastListBasedThingie knows its own invariants and it knows that the regular list input is an exact list with no special invariants). See http://en.wikipedia.org/wiki/GRASP_(Object_Oriented_Design) for the InformationExpert pattern. The other advantage is that it leaves the API for general purpose lists completely unmolested. That API needs to be as friendly as possible. [Kristj?n Valur J?nsson] > There is precedence, I think. > list.reverse() is semantically identical to > list[:] = reversed(list) It's not identical. There is no list copy or mutation. Instead it returns either a custom list iterator provided by the listobject itself (which knows its own invariants) or it returns an iterator that calls the list's own __getitem__ method. There is no danger to the target of reversed(). > Same goes for sort(), really. The sort() method is closer in spirit. It does mutate the list in-place, so any list subclasses must either disable sort() by overriding it or it needs to limit its invariants to ones that aren't affected by the mutation. This is unfortunate because it places a burden on every class that inherits from list. So even though you've found a precedent, I don't think it is something that we want to have more of. Also, the situation for swap() is made more difficult because there are two parties (the swapper and the swappee). With sort(), a subclass is giving permission for mutation by not overriding the sort method. With swap(), the swappee has no say in the matter. There is no way for a swappee's class to declare that its invariants do not support swapping. The situation is even more acute when the swappee is a C extension inheriting from list -- violating its invariants may crash the interpreter. Raymond From kristjan at ccpgames.com Fri Jun 26 00:37:36 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 25 Jun 2009 22:37:36 +0000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local><3c722d20-fa00-4d19-a356-6333eeec2d07@z4g2000prh.googlegroups.com><930F189C8A437347B80DF2C156F7EC7F057FB5FBB0@exchis.ccp.ad.local><930F189C8A437347B80DF2C156F7EC7F057FB5FDAA@exchis.ccp.ad.local> <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> Message-ID: <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> Since this is Python-ideas, I'm going to persist a bit with my argumentation, even though I concede defeat :) I'm going for a generic SwapLists(a,b) function in C, since my list subclass isn't written in C. > -----Original Message----- > From: Raymond Hettinger [mailto:python at rcn.com] > Sent: 25. j?n? 2009 21:58 > > The sort() method is closer in spirit. It does mutate the list in- > place, so any list subclasses must either disable sort() by > overriding it or it needs to limit its invariants to ones that aren't > affected by the mutation. This is unfortunate because it > places a burden on every class that inherits from list. So even though > you've found a precedent, I don't think it is something that > we want to have more of. Since I started uncovering list bad-boys, I think perhaps list.pop() might be the evilest of the bunch in this context, since it does change the list length and so might crash unwary C extensions with some invariants. > > Also, the situation for swap() is made more difficult because there are > two parties (the swapper and the swappee). With sort(), a > subclass is giving permission for mutation by not overriding the sort > method. With swap(), the swappee has no say in the matter. > There is no way for a swappee's class to declare that its invariants do > not support swapping. The situation is even more acute when > the swappee is a C extension inheriting from list -- violating its > invariants may crash the interpreter. I do have a suggested solution for that: Require that the swapee is not a list subclass, by using PyList_CheckExact(). That does make it safer at the cost of narrowing the application domain. Anyway, I'm going to mark the issue as "rejected" and get on with life. Thank you Raymond for your insightful comments. K From steve at pearwood.info Fri Jun 26 02:41:29 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 26 Jun 2009 10:41:29 +1000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> Message-ID: <200906261041.30075.steve@pearwood.info> On Fri, 26 Jun 2009 08:37:36 am Kristj?n Valur J?nsson wrote: > Since I started uncovering list bad-boys, I think perhaps list.pop() > might be the evilest of the bunch in this context, since it does > change the list length and so might crash unwary C extensions with > some invariants. True, the caller can shoot himself in the foot by bypassing your class and calling list.pop(instance) instead of instance.pop(). That's a bad thing, but probably unavoidable given Python's OO design. However, it's also a fairly unusual thing to do: like calling "private" double-underscore methods, anyone calling superclass.method(instance) instead of instance.method() will know that they're doing something unsupported and potentially dangerous. Since we agree it's a bad thing, why do you want to copy it and create another method which is even more dangerous, by design? As I see it, Python already has a perfectly good solution for swapping the contents of arbitrary mutable sequences, one which doesn't break any invariants: a[:], b[:] = b[:], a[:] What you really want is a fast way of initialising a list, given another list. That was your use-case, after all. Swapping the contents is just one particular implementation of that fast-initialise. I believe that the correct place for this is inside a list constructor, not as an external method which magically transports the internal data from one list into another. I suggested adding extra functionality to list.__new__, and Nick suggested a new constructor list.from_list(). Nick suggested making from_list destructive, but I don't see the advantage of doing so. I think this is a probably a more productive direction to take than a swap() method. -- Steven D'Aprano From python at rcn.com Fri Jun 26 05:36:43 2009 From: python at rcn.com (Raymond Hettinger) Date: Thu, 25 Jun 2009 20:36:43 -0700 Subject: [Python-ideas] add a list.swap() method References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local><1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1><930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> <200906261041.30075.steve@pearwood.info> Message-ID: [Steven D'Aprano] > I suggested adding extra functionality to > list.__new__, and Nick suggested a new constructor list.from_list(). > Nick suggested making from_list destructive, but I don't see the > advantage of doing so. I FWIW, the constructor for list objects already has a fast path for copying an input list or tuple. So the only new thing you can add is a destructive, steal the pointer approach that breaks encapsulation and opens a can of worms. Raymond From ncoghlan at gmail.com Fri Jun 26 11:06:13 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Jun 2009 19:06:13 +1000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <200906261041.30075.steve@pearwood.info> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> <200906261041.30075.steve@pearwood.info> Message-ID: <4A448F85.9090509@gmail.com> Steven D'Aprano wrote: > On Fri, 26 Jun 2009 08:37:36 am Kristj?n Valur J?nsson wrote: > >> Since I started uncovering list bad-boys, I think perhaps list.pop() >> might be the evilest of the bunch in this context, since it does >> change the list length and so might crash unwary C extensions with >> some invariants. > > True, the caller can shoot himself in the foot by bypassing your class > and calling list.pop(instance) instead of instance.pop(). That's a bad > thing, but probably unavoidable given Python's OO design. I think Kristj?n's point was that list.pop() and friends don't perform a PyList_CheckExact on themselves before following their fast path implementation so they can bypass custom get/set/del methods in subclasses. So subclasses that override __delitem__ to do some additional bookkeeping without overriding pop() could get a nasty surprise when someone calls pop() on them. This would be similar to the oddities that can happen when you override dict.__setitem__ without overriding dict.update(). Then again, maybe we just need to document this behaviour as a trap of subclassing the mutable builtins: make sure to override *all* the mutating methods in subclasses, not just the special methods. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From kristjan at ccpgames.com Fri Jun 26 14:05:23 2009 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Fri, 26 Jun 2009 12:05:23 +0000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <4A448F85.9090509@gmail.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> <200906261041.30075.steve@pearwood.info> <4A448F85.9090509@gmail.com> Message-ID: <930F189C8A437347B80DF2C156F7EC7F057FD17B67@exchis.ccp.ad.local> > -----Original Message----- > From: python-ideas-bounces+kristjan=ccpgames.com at python.org > [mailto:python-ideas-bounces+kristjan=ccpgames.com at python.org] On > Behalf Of Nick Coghlan > Sent: 26. j?n? 2009 09:06 > To: Steven D'Aprano > Cc: python-ideas at python.org > Subject: Re: [Python-ideas] add a list.swap() method > > > I think Kristj?n's point was that list.pop() and friends don't perform > a > PyList_CheckExact on themselves before following their fast path > implementation so they can bypass custom get/set/del methods in > subclasses. Right. I've already agreed that my suggestion idea is too radical. I'm merely observing that our discussion has pointed out that other list member functions can be dangerous for list subclasses since they too perform "magic shortcuts" to achieve their goal. So, those that write list subclasses that maintain internal state based on their members, must be careful to override all of these magic list functions. They are list.append, list.pop, list.reverse, list.sort, list.insert, that I can see with a casual perusal of the source. In particular, append(), pop() and insert() modify the list length, which might confuse some subclasses. Cheers, K From python at rcn.com Fri Jun 26 17:57:55 2009 From: python at rcn.com (Raymond Hettinger) Date: Fri, 26 Jun 2009 08:57:55 -0700 Subject: [Python-ideas] add a list.swap() method References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local><1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1><930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local><200906261041.30075.steve@pearwood.info> <4A448F85.9090509@gmail.com> <930F189C8A437347B80DF2C156F7EC7F057FD17B67@exchis.ccp.ad.local> Message-ID: <830F54C254564D3C8FDE7369A7C52328@RaymondLaptop1> [Kristj?n Valur J?nsson] > In particular, append(), pop() and insert() modify the list length, which might confuse some subclasses. These are all part of the most basic functions of list. A subclasser should know that. They are different from swap() which is an optimization hack that breaks encapsulation and is not a basic function of lists. The docs are fine as-is. Adding esoterica to the docs tends to make the docs harder to digest and less useful to people trying to learn the language. Raymond From tjreedy at udel.edu Fri Jun 26 22:41:30 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 26 Jun 2009 16:41:30 -0400 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <830F54C254564D3C8FDE7369A7C52328@RaymondLaptop1> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local><1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1><930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local><200906261041.30075.steve@pearwood.info> <4A448F85.9090509@gmail.com> <930F189C8A437347B80DF2C156F7EC7F057FD17B67@exchis.ccp.ad.local> <830F54C254564D3C8FDE7369A7C52328@RaymondLaptop1> Message-ID: Raymond Hettinger wrote: > > [Kristj?n Valur J?nsson] >> In particular, append(), pop() and insert() modify the list length, >> which might confuse some subclasses. > > These are all part of the most basic functions of list. I have to chuckle a bit at that statement. When I posted the proposal to add .pop() (before there were formal 'PEP's), a couple of people accused my of trying to ruin Python. So agree, even if they did not, and I am glad you think so too ;-) > A subclasser should know that. > They are different from swap() which is an optimization hack that breaks > encapsulation and is not a basic function of lists. The docs are fine as-is. > > Adding esoterica to the docs tends to make the docs harder to digest > and less useful to people trying to learn the language. As someone who hopes to be introducing Python to new people, I appreciate that you keep digestibility in mind, even if I occasionally forget in the quest for technical accuracy (which is also important). Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sat Jun 27 02:25:58 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 27 Jun 2009 12:25:58 +1200 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <4A448F85.9090509@gmail.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> <200906261041.30075.steve@pearwood.info> <4A448F85.9090509@gmail.com> Message-ID: <4A456716.6010406@canterbury.ac.nz> Nick Coghlan wrote: > Then again, maybe we just need to document this behaviour as a trap of > subclassing the mutable builtins: make sure to override *all* the > mutating methods in subclasses, not just the special methods. I would never have imagined things were otherwise. This is a general principle that applies to all classes, whether builtin or not: unless explicitly documented, you can't make any assumptions about how methods call each other internally. -- Greg From ncoghlan at gmail.com Sat Jun 27 09:59:09 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 27 Jun 2009 17:59:09 +1000 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <830F54C254564D3C8FDE7369A7C52328@RaymondLaptop1> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local><1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1><930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local><200906261041.30075.steve@pearwood.info> <4A448F85.9090509@gmail.com> <930F189C8A437347B80DF2C156F7EC7F057FD17B67@exchis.ccp.ad.local> <830F54C254564D3C8FDE7369A7C52328@RaymondLaptop1> Message-ID: <4A45D14D.8050603@gmail.com> Raymond Hettinger wrote: > > [Kristj?n Valur J?nsson] >> In particular, append(), pop() and insert() modify the list length, >> which might confuse some subclasses. > > These are all part of the most basic functions of list. A subclasser > should know that. > They are different from swap() which is an optimization hack that breaks > encapsulation > and is not a basic function of lists. The docs are fine as-is. > > Adding esoterica to the docs tends to make the docs harder to digest > and less useful to people trying to learn the language. I was thinking more in terms of the C API docs rather than the normal list docs. While Python subclasses can run into bugs due to this they're unlikely to crash the interpreter by doing so. C subclasses on the other hand may have bigger problems. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tn.pablo at gmail.com Mon Jun 29 17:48:53 2009 From: tn.pablo at gmail.com (Pablo Torres N.) Date: Mon, 29 Jun 2009 10:48:53 -0500 Subject: [Python-ideas] Make all switches keyword-only Message-ID: Hello! This is my first time posting to a python list, so be kind with the newbie :-) >From issue 6372: "I propose that all formal parameters that really act as options/switches be made keyword-only. Examples of switches are all flags, timeouts, 'verbose' bools, maximums and minimums, etc. This stresses the difference between needed input for a function and an argument that changes/extends the behavior. Besides, the code would be more readable, because instead of having some cryptic function call like register('Pablo Torres', 2, 4, 5, False) you would have the prettier register('Pablo Torres', hour=2, min=4, sec=5, had_reservation=False). The implementation would just require putting a star '*' before all options, according to pep 3102. If needed, I can rewrite this as a PEP." Since this would break pretty much all the code out there, I suspect that we should come up with a warning mechanism first. Maybe leave the warning there for a couple of releases before making it definitive. What do you guys think? From jared.grubb at gmail.com Mon Jun 29 18:57:47 2009 From: jared.grubb at gmail.com (Jared Grubb) Date: Mon, 29 Jun 2009 09:57:47 -0700 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <200906261041.30075.steve@pearwood.info> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> <200906261041.30075.steve@pearwood.info> Message-ID: <433DDD4F-4D62-45AB-BDC2-5D864F2E7B53@gmail.com> On 25 Jun 2009, at 17:41, Steven D'Aprano wrote: > What you really want is a fast way of initialising a list, given > another > list. That was your use-case, after all. Swapping the contents is just > one particular implementation of that fast-initialise. FWIW, this use case really sounds a lot like rvalue references, a new feature being added to the new C++0x standard. The basic idea is that, for efficiency reasons, you want to be able to "sink" the contents an object and leave the object in a destructable (but not necessarily valid) state. This would be a tough (and probably inappropriate?) feature in Python, but I mention it because the concept might provide further context/ justifications/trade-offs/ideas for what is being discussed here. Jared From tjreedy at udel.edu Mon Jun 29 20:11:22 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 29 Jun 2009 14:11:22 -0400 Subject: [Python-ideas] Make all switches keyword-only In-Reply-To: References: Message-ID: Pablo Torres N. wrote: > Hello! This is my first time posting to a python list, so be kind with > the newbie :-) You propose to break nearly everyone's code and make writing Python code harder by having your stylistic preference imposed on everyone. Do you consider *that* to be 'kind'? Before proposing wholesale changes, I kindly suggest that you please think about and learn the following Python design and principles: Python is a consenting-adults language. Breaking existing code is bad, and should only be done with excellent reason. >>From issue 6372: > > "I propose that all formal parameters that really act as > options/switches be made keyword-only. Examples of switches are all > flags, timeouts, 'verbose' bools, maximums and minimums, etc. Once you get beyond binary switches, you enter the land of ambiguity and endless argument. > This stresses the difference between needed input for a function and an > argument that changes/extends the behavior. Besides, the code would be > more readable, because instead of having some cryptic function call like > register('Pablo Torres', 2, 4, 5, False) you would have the prettier > register('Pablo Torres', hour=2, min=4, sec=5, had_reservation=False). You are free to write your code like this. Do you? How many 100s or 1000s of lines? This style require that one remember the *exact* name of each keyword parameter, rather than just the position. Easier for you, perhaps, but not for everyone. It also requires that one accurately type 'name ='. More chance for typos. > The implementation would just require putting a star '*' before all > options, according to pep 3102. You are free to write your functions like this. Others are free to refuse to use them if you do. > If needed, I can rewrite this as a PEP." Please do not. > Since this would break pretty much all the code out there, I suspect > that we should come up with a warning mechanism first. Maybe leave the > warning there for a couple of releases before making it definitive. People are already not switching to Python3 because it breaks existing code. Others who would like to, cannot, because the 3rd-party libraries they need have not been converted. There should not be any more wholesale code break changes in Python 3. > What do you guys think? You should have posted to python-list something like the following. subj: Arguments by keyword. "I like this style of calling functions. What do you guys think?" Terry Jan Reedy From jimjjewett at gmail.com Mon Jun 29 23:33:02 2009 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 29 Jun 2009 17:33:02 -0400 Subject: [Python-ideas] Make all switches keyword-only In-Reply-To: References: Message-ID: On Mon, Jun 29, 2009 at 11:48 AM, Pablo Torres N. wrote: > Hello! ?This is my first time posting to a python list, so be kind with > the newbie :-) > > >From issue 6372: > > "I propose that all formal parameters that really act as > options/switches be made keyword-only. It is probably way too late for existing functions. What you might propose (but it would still be a long shot) are changes to PEP 8, which often gets used as a style guide. You may wish to recommend that (a) *future* such functions be keyword-only, and (b) current calls be updated to act as though the underlying functions were keyword-only. -jJ From tn.pablo at gmail.com Mon Jun 29 23:48:53 2009 From: tn.pablo at gmail.com (Pablo Torres N.) Date: Mon, 29 Jun 2009 16:48:53 -0500 Subject: [Python-ideas] Make all switches keyword-only In-Reply-To: References: Message-ID: > It is probably way too late for existing functions. ?What you might > propose (but it would still be a long shot) are changes to PEP 8, > which often gets used as a style guide. > > You may wish to recommend that > (a) ?*future* such functions be keyword-only, and > (b) ?current calls be updated to act as though the underlying > functions were keyword-only. > > -jJ > After all the bad critics here and in the bug tracker, I think I'll leave it to that. From Scott.Daniels at Acm.Org Tue Jun 30 00:22:25 2009 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Mon, 29 Jun 2009 15:22:25 -0700 Subject: [Python-ideas] add a list.swap() method In-Reply-To: <433DDD4F-4D62-45AB-BDC2-5D864F2E7B53@gmail.com> References: <930F189C8A437347B80DF2C156F7EC7F057FB5F90B@exchis.ccp.ad.local> <1D23A2AE52FD45FA823818DDB0C32D8F@RaymondLaptop1> <930F189C8A437347B80DF2C156F7EC7F057FB5FDC0@exchis.ccp.ad.local> <200906261041.30075.steve@pearwood.info> <433DDD4F-4D62-45AB-BDC2-5D864F2E7B53@gmail.com> Message-ID: Jared Grubb wrote: > > On 25 Jun 2009, at 17:41, Steven D'Aprano wrote: >> What you really want is a fast way of initialising a list, given another >> list. That was your use-case, after all. Swapping the contents is just >> one particular implementation of that fast-initialise. > > FWIW, this use case really sounds a lot like rvalue references, a new > feature being added to the new C++0x standard. The basic idea is that, > for efficiency reasons, you want to be able to "sink" the contents an > object and leave the object in a destructable (but not necessarily > valid) state. > > This would be a tough (and probably inappropriate?) feature in Python, > but I mention it because the concept might provide further > context/justifications/trade-offs/ideas for what is being discussed here. The OP might consider code that does the swap if the refcount is 1, otherwise does something akin to recur(arg.copy()) (which _will_ have a refcount of 0). --Scott David Daniels Scott.Daniels at Acm.Org