From Nikolaus at rath.org Thu Oct 1 00:42:10 2015 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 30 Sep 2015 15:42:10 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <653DC9A1-3C9B-4E66-AA97-4CA24F21F9E9@yahoo.com> (Andrew Barnert via Python-ideas's message of "Wed, 30 Sep 2015 14:40:15 -0700") References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560B1E49.7050102@canterbury.ac.nz> <560C12AD.90305@trueblade.com> <653DC9A1-3C9B-4E66-AA97-4CA24F21F9E9@yahoo.com> Message-ID: <877fn78r7h.fsf@thinkpad.rath.org> On Sep 30 2015, Andrew Barnert via Python-ideas wrote: > Also, explicit "it" would be usable in other situations: > > z = dangerous_thing(arg) if it.value() > 3 else DummyValue(3) Really? I'm not sure if "it" is "dangerous_thing" or "dangerous_thing(arg)". Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From srkunze at mail.de Thu Oct 1 00:57:52 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 01 Oct 2015 00:57:52 +0200 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560BC0C3.4040502@btinternet.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560B1E49.7050102@canterbury.ac.nz> <560BC0C3.4040502@btinternet.com> Message-ID: <560C68F0.4080600@mail.de> On 30.09.2015 13:00, Rob Cliffe wrote: > Or: > x = a orelse b # Visual Basic has a short-circuiting OrElse > operator for boolean operands > x = a orifNone b > > > On 30/09/2015 01:39, Ryan Gonzalez wrote: >> What about 'otherwise'? >> >> x = a otherwise b >> The only reason why I would prefer "else" over the proposed alternatives: it's already a reserved keyword and it's not really necessary to waste another one. Otherwise, I don't care too much. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Thu Oct 1 01:07:27 2015 From: random832 at fastmail.com (Random832) Date: Wed, 30 Sep 2015 19:07:27 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <877fn78r7h.fsf@thinkpad.rath.org> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560B1E49.7050102@canterbury.ac.nz> <560C12AD.90305@trueblade.com> <653DC9A1-3C9B-4E66-AA97-4CA24F21F9E9@yahoo.com> <877fn78r7h.fsf@thinkpad.rath.org> Message-ID: <1443654447.723246.398079841.55A7B687@webmail.messagingengine.com> On Wed, Sep 30, 2015, at 18:42, Nikolaus Rath wrote: > On Sep 30 2015, Andrew Barnert via Python-ideas > wrote: > > Also, explicit "it" would be usable in other situations: > > > > z = dangerous_thing(arg) if it.value() > 3 else DummyValue(3) > > Really? I'm not sure if "it" is "dangerous_thing" or > "dangerous_thing(arg)". I thought it was arg, though it's a bit silly for it to be. Why not go full Scheme and add a way to do lexical expressions that capture values for multiple use: z = (let it=arg: dangerous_thing(it) if it.value() > 3 else DummyValue(3)) Would be equivalent to z = (lambda it: dangerous_thing(it) if it.value() > 3 else DummyValue(3))(arg) Only without necessarily actually instantiating a lambda (i.e. it could push the result of arg [which may be any expression] onto the stack or in an anonymous or obscurely-named local variable slot to refer whenever it is referenced in the inner expression. From srkunze at mail.de Thu Oct 1 01:15:03 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 01 Oct 2015 01:15:03 +0200 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> Message-ID: <560C6CF7.10609@mail.de> On 30.09.2015 23:33, Andrew Barnert via Python-ideas wrote: > But you're conflating the concept of "lazy" with the concept of "iterator". While generators, and iterators in general, are always technically lazy and nearly-always practically lazy, lazy things are not always iterators. Range, dict views, memoryview/buffer objects, NumPy slices, third-party lazy-list types, etc. are not generators, nor are they like generators in any way, except for being lazy. They're lazy sequences (well, except for the ones that aren't sequences, but they're still lazy containers, or lazy non-iterator iterables if you want to stick to terms in the glossary). > > And I think experienced developers conflating the two orthogonal concepts is part of what leads to novices getting confused. They think that if they want laziness, they need a generator. That makes them unable to even form the notion that what they really want is a view/lazy container/virtual container even when that's what they want. > > And it makes it hard to discuss issues like this thread clearly. > > (The fact that we don't have a term for "non-iterator iterable", and that experienced users and even the documentation sometimes use the term "sequence" for that, only makes things worse. For example, a dict_keys is not a sequence in any useful sense, but the glossary says it is, because there is no word for what it wants to say.) I have absolutely no idea what you are talking about here. ;) I have to admit I try to avoid thinking too much about such tiny little details by using generators/lists/sequences/iterables/iterators/did-I-miss-one? directly in for loops only. Thus, the differences between all of them go away pretty fast. But honestly, does it really need to be that complicated? Best, Sven From tjreedy at udel.edu Thu Oct 1 02:04:26 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 30 Sep 2015 20:04:26 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> Message-ID: On 9/30/2015 5:33 PM, Andrew Barnert via Python-ideas wrote: > (The fact that we don't have a term for "non-iterator iterable", 'collection' Some are concrete: they contain reference to actual Python objects. Some are virtual (lazy): they contain the information need to create Python objects as needed. Strings are a bit in between. -- Terry Jan Reedy From 4kir4.1i at gmail.com Thu Oct 1 02:28:13 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 01 Oct 2015 03:28:13 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> Message-ID: <87vbar300y.fsf@gmail.com> Andrew Barnert via Python-ideas writes: ... > (The fact that we don't have a term for "non-iterator iterable", and All iterators are iterable but some iterables are not iterators. If your code accepts only iterators then use the term *iterator*. Otherwise the term *iterable* could be used. It is misleading to use *iterable* if your code only accepts iterators. If an iterable is an iterator; It is called *iterator*. The term *iterable* implies that some instances are not iterators. From steve at pearwood.info Thu Oct 1 02:33:58 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 1 Oct 2015 10:33:58 +1000 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560B1E49.7050102@canterbury.ac.nz> Message-ID: <20151001003358.GA23642@ando.pearwood.info> On Wed, Sep 30, 2015 at 09:41:53AM -0700, Jeff Hardy wrote: > 'def' is currently short for 'define', which would be too confusing. > Spelling out 'default' isn't so bad, though: > > self.x = x default [] Heh, people jest :-) Oh, you're serious? That just goes to show how different we all are. I think that using "default" as an infix operator is hideous. "default" as a function/verb? Sure, that works fine: if len(missed_payments) > 5: loan.default() "default" as a variable/noun? Absolutely fine: default = "something or other" But "x default z" just doesn't read well: in English, it sounds like you are ordering x to cause z to default. Not very many words read naturally as a binary infix operator. Besides, I would expect that making "default" a keyword will break a *huge* number of programs. I'm sure that many, many people use it as a variable or parameter name. > And if it's going to be that long anyway, we might as well just put a > `default` function in the builtins: > > self.x = default(x, []) That's useless, as it breaks the short-circuiting property, which is one of the critical motives for introducing the null coalescing operator. Without the short-circuiting property, there's no point in making it a built-in. If all you want is a default function, then just add it as a helper function to the top of your code: def default(obj, alternative): if obj is None: return alternative return obj > I actually really like 'otherwise', but it's certainly not brief: > > self.x = x if x is not None else [] > self.x = x otherwise [] That's almost half the length of the alternative. And if you have something more realistic, you save proportionally even less: document if document is not None else Document() document otherwise Document() But personally, I have absolutely zero interest in the null coalescing operator on its own. I don't object to it specifically, but what really interests me are the two (pseudo)operators, null-aware indexing and null-aware attribute lookup. How do you extend "otherwise" to work with the [] and . operators? -- Steve From random832 at fastmail.com Thu Oct 1 02:46:48 2015 From: random832 at fastmail.com (Random832) Date: Wed, 30 Sep 2015 20:46:48 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> Message-ID: Akira Li <4kir4.1i at gmail.com> writes: > Andrew Barnert via Python-ideas > writes: > ... >> (The fact that we don't have a term for "non-iterator iterable", and > > All iterators are iterable but some iterables are not iterators. > > If your code accepts only iterators then use the term *iterator*. > Otherwise the term *iterable* could be used. > > It is misleading to use *iterable* if your code only accepts iterators. > > If an iterable is an iterator; It is called *iterator*. The term > *iterable* implies that some instances are not iterators. There are three (well, three and a half) kinds of code that consume iterables, how would you describe each simply? 1. Does not call iter, simply calls next. Therefore cannot consume a non-iterator iterable. 2. Calls iter, but can accept an iterator (e.g. only goes through it once) 3. Cannot accept an iterator (goes through it twice, or permanently stores a reference to it, etc) 4. Can accept either, but behaves differently in each case (e.g. zip when passed two of the same iterator) - this can be regarded as a special case of #2. From python at mrabarnett.plus.com Thu Oct 1 03:08:51 2015 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 1 Oct 2015 02:08:51 +0100 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> Message-ID: <560C87A3.1020507@mrabarnett.plus.com> On 2015-09-29 21:43, Jeff Hardy wrote: > On Tue, Sep 29, 2015 at 10:35 AM, Barry Warsaw > wrote: > > On Sep 28, 2015, at 03:04 PM, Carl Meyer wrote: > > >But even if they are rejected, I think a simple `??` or `or?` (or > >however it's spelled) operator to reduce the repetition of "x if x is > >not None else y" is worth consideration on its own merits. This operator > >is entirely unambiguous, and I think would be useful and frequently > >used, whether or not ?. and ?[ are added along with it. > > But why is it an improvement? The ternary operator is entirely > obvious and > readable, and at least in my experience, is rare enough that the > repetition > doesn't hurt my fingers that much. It seems like such a radical, > ugly new > syntax unjustified by the frequency of use and readability improvement. > > > I use it all over the place in C# code, where it makes null checks much > cleaner, and the punctuation choice makes sense: > > var x = foo != null ? foo : ""; > var y = foo ?? ""; > > (it also has other uses in C# relating to nullable types that aren't > relevant in Python.) > > I'd argue the same is true in Python, if a decent way to spell it can be > found: > > x = foo if foo is not None else "" > y = foo or? "" > > It's pure syntactic sugar, but it *is* pretty sweet. > > (It would also make get-with-default unnecessary, but since it already > exists that's not a useful argument.) > It's only just occurred to me that there's a small inconsistency here. The "?.", "?[" and "?(" will short-circuit on None, whereas the "??" will short-circuit on non-None. Would that cause any confusion in practice? From steve at pearwood.info Thu Oct 1 03:15:06 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 1 Oct 2015 11:15:06 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: Message-ID: <20151001011504.GB23642@ando.pearwood.info> On Wed, Sep 30, 2015 at 05:19:53PM +0000, Neil Girdhar wrote: > I guess, I'm just asking for enumerate to go through the same change that > range went through. Why wasn't it a problem for range? There is a pernicious myth that (x)range is an iterator. It is not. It is a sequence, but one where the items are calculated on demand rather than pre-populated into some large data structure (a list or array). This is not just a matter of labels. It is a matter of the actual behaviour. (x)range objects don't behave like iterators except in the simplest sense that you can iterate over them. So your question is based on false assumptions -- range didn't go through any such change. In Python 2, range was eager and xrange lazy, but both are sequences, and in Python 3 the eager version is gone and the lazy version renamed without the "x" prefix. -- Steve From steve at pearwood.info Thu Oct 1 03:31:06 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 1 Oct 2015 11:31:06 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> Message-ID: <20151001013106.GC23642@ando.pearwood.info> On Wed, Sep 30, 2015 at 12:19:05PM -0700, Andrew Barnert via Python-ideas wrote: [...] > > FWIW: I don't think many people use the lazy sequence features > > of range(), e.g. the slicing or index support. By far most > > uses are in for-loops. > > I've used range as a sequence (or at least a reusable iterable, a > sized object, and a container). I've answered questions from people on > StackOverflow who are doing so, and seen the highest-rep Python > answerer on SO suggest such uses to other people. > > I don't think I'd ever use the index method (although I did see one SO > user who was doing so, to wrap up some arithmetic in a way that avoids > a possibly off-by-one error, and wanted to know why it was so slow in > 3.1 but worked fine in 3.2...), but there's no reason range should be > a defective "not-quite-sequence" instead of a sequence. What would be > the point of that? There's also __contains__. Personally, I don't like it, but using "n in range(a, b+1)" for testing whether integer n falls within a particular range seems to be popular. I don't know why they don't just write a <= n <= b, but it seems to be a popular idiom for some weird reason. -- Steve From 4kir4.1i at gmail.com Thu Oct 1 04:04:07 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 01 Oct 2015 05:04:07 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> Message-ID: <87r3lf2vl4.fsf@gmail.com> Random832 writes: > Akira Li <4kir4.1i at gmail.com> writes: > >> Andrew Barnert via Python-ideas >> writes: >> ... >>> (The fact that we don't have a term for "non-iterator iterable", and >> >> All iterators are iterable but some iterables are not iterators. >> >> If your code accepts only iterators then use the term *iterator*. >> Otherwise the term *iterable* could be used. >> >> It is misleading to use *iterable* if your code only accepts iterators. >> >> If an iterable is an iterator; It is called *iterator*. The term >> *iterable* implies that some instances are not iterators. > > There are three (well, three and a half) kinds of code that consume > iterables, how would you describe each simply? > > 1. Does not call iter, simply calls next. Therefore cannot consume a > non-iterator iterable. iterator > 2. Calls iter, but can accept an iterator (e.g. only goes through it > once) iterable > 3. Cannot accept an iterator (goes through it twice, or permanently > stores a reference to it, etc) neither *iterable* is an object that you can pass to iter() to get *iterator*. An iterable does not guarantee that it yields the same items twice. > 4. Can accept either, but behaves differently in each case (e.g. zip > when passed two of the same iterator) - this can be regarded as a > special case of #2. iterable From abarnert at yahoo.com Thu Oct 1 04:24:03 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 19:24:03 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> Message-ID: <27D77A4D-7CDE-470D-BD55-AC25CE9F782B@yahoo.com> On Sep 30, 2015, at 17:04, Terry Reedy wrote: > >> On 9/30/2015 5:33 PM, Andrew Barnert via Python-ideas wrote: >> >> (The fact that we don't have a term for "non-iterator iterable", > > 'collection' That's a perfectly good term, but it's not used that way in the docs, nor is anyone else using it in the discussions so far. Are you suggesting that we should start doing so? There are definitely parts of the docs that could be clarified or simplified with this term, such as the glossary entry and definitions for dict views, which inaccurately use the term "sequence". (And similarly, although not quite as badly, someone in this thread referred to "sequences and sequence-like things", which may be a little more intuitive than my "non-iterator iterables", but still isn't all that clear.) Also, the tutorial uses the phrases "data structures" or "data type" a few zillion times, apparently to avoid having to come up with a term that includes sequences, sets, dicts, and strings without being inaccurate. I've seen novices have no idea what "data structure" means, or get confused by what the difference between a "data type" and a "regular type" is. > Some are concrete: they contain reference to actual Python objects. > Some are virtual (lazy): they contain the information need to create Python objects as needed. I think the docs used to use the word "virtual" as a more specific term than "lazy": a view onto an object that conceptually exists but doesn't actually exist is "virtual" (like range, which is a view into the infinite set of integers), but a view into a real object isn't (like dict_keys, which has a reference to an actual dict), nor is something that isn't conceptually view-like at all (like a RNG iterator), even though they're all "lazy". It looks like the word "virtual" in this context doesn't appear anywhere in the docs anymore, so I suppose it could be repurposed, but if it's just a synonym for "lazy", what's wrong with "lazy"? From rob.cliffe at btinternet.com Thu Oct 1 04:08:47 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 1 Oct 2015 03:08:47 +0100 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560C68F0.4080600@mail.de> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560B1E49.7050102@canterbury.ac.nz> <560BC0C3.4040502@btinternet.com> <560C68F0.4080600@mail.de> Message-ID: <560C95AF.1020501@btinternet.com> On 30/09/2015 23:57, Sven R. Kunze wrote: > On 30.09.2015 13:00, Rob Cliffe wrote: >> Or: >> x = a orelse b # Visual Basic has a short-circuiting >> OrElse operator for boolean operands >> x = a orifNone b >> >> >> On 30/09/2015 01:39, Ryan Gonzalez wrote: >>> What about 'otherwise'? >>> >>> x = a otherwise b >>> > > > The only reason why I would prefer "else" over the proposed > alternatives: it's already a reserved keyword and it's not really > necessary to waste another one. Otherwise, I don't care too much. > > Best, > Sven Are you suggesting that "a else b" is a possibility? That's a no-no because a if b else c else d is ambiguous; it could mean either of a if b else (c else d) (a if b else c) else d Rob > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Oct 1 04:30:02 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 30 Sep 2015 21:30:02 -0500 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560C95AF.1020501@btinternet.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560B1E49.7050102@canterbury.ac.nz> <560BC0C3.4040502@btinternet.com> <560C68F0.4080600@mail.de> <560C95AF.1020501@btinternet.com> Message-ID: On September 30, 2015 9:08:47 PM CDT, Rob Cliffe wrote: > > >On 30/09/2015 23:57, Sven R. Kunze wrote: >> On 30.09.2015 13:00, Rob Cliffe wrote: >>> Or: >>> x = a orelse b # Visual Basic has a short-circuiting >>> OrElse operator for boolean operands >>> x = a orifNone b >>> >>> >>> On 30/09/2015 01:39, Ryan Gonzalez wrote: >>>> What about 'otherwise'? >>>> >>>> x = a otherwise b >>>> >> >> >> The only reason why I would prefer "else" over the proposed >> alternatives: it's already a reserved keyword and it's not really >> necessary to waste another one. Otherwise, I don't care too much. >> >> Best, >> Sven >Are you suggesting that "a else b" is a possibility? >That's a no-no because > a if b else c else d >is ambiguous; it could mean either of > a if b else (c else d) > (a if b else c) else d >Rob > I already said that... >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From abarnert at yahoo.com Thu Oct 1 04:31:23 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 19:31:23 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <87r3lf2vl4.fsf@gmail.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> Message-ID: On Sep 30, 2015, at 19:04, Akira Li <4kir4.1i at gmail.com> wrote: > > Random832 writes: > >> Akira Li <4kir4.1i at gmail.com> writes: >> >>> Andrew Barnert via Python-ideas >>> writes: >>> ... >>>> (The fact that we don't have a term for "non-iterator iterable", and >>> >>> All iterators are iterable but some iterables are not iterators. >>> >>> If your code accepts only iterators then use the term *iterator*. >>> Otherwise the term *iterable* could be used. >>> >>> It is misleading to use *iterable* if your code only accepts iterators. >>> >>> If an iterable is an iterator; It is called *iterator*. The term >>> *iterable* implies that some instances are not iterators. >> >> There are three (well, three and a half) kinds of code that consume >> iterables, how would you describe each simply? >> >> 1. Does not call iter, simply calls next. Therefore cannot consume a >> non-iterator iterable. > iterator > >> 2. Calls iter, but can accept an iterator (e.g. only goes through it >> once) > iterable > >> 3. Cannot accept an iterator (goes through it twice, or permanently >> stores a reference to it, etc) > neither > > *iterable* is an object that you can pass to iter() to get *iterator*. > An iterable does not guarantee that it yields the same items twice. And this is exactly the problem. We don't have any way to simply describe this thing. Hence all the confusion in this thread, and in similar discussions elsewhere, and even in the docs (e.g., describing dict views as sequences and then going on to explain that they're not actually sequences). The fact that it took your previous message four paragraphs without inventing a new term, to say what I said in one sentence with a new term, demonstrates the problem. As does the fact that my attempted new term, "non-iterator iterable", is sufficiently ugly and not intuitively helpful enough that you felt the need to expand on it for four paragraphs. From abarnert at yahoo.com Thu Oct 1 04:39:13 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 19:39:13 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560C87A3.1020507@mrabarnett.plus.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> Message-ID: <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> On Sep 30, 2015, at 18:08, MRAB wrote: > > It's only just occurred to me that there's a small inconsistency here. > The "?.", "?[" and "?(" will short-circuit on None, whereas the "??" > will short-circuit on non-None. > > Would that cause any confusion in practice? I noticed this when I was trying to write out grammar, sample ASTs, and sample bytecode for these things. I went searching the thread and saw no one had pointed it out. I went through docs and blogs for other languages, and didn't see anyone pointing out, complaining about, or offering to clear up any confusion. So I figured I wouldn't mention it, and see if anyone else even noticed. The fact that an experienced programmer like you didn't even notice it until after a zillion messages over a span of weeks, and apparently nobody else did at all, seems to imply that there's no dangerously misleading intuitive parallel here. (By the way, I have a feeling that including ?= will increase the risk of confusion, but I have no idea where that feeling comes from, so it may just be random noise in my head, maybe because I like ?. but don't particularly like ?= or something...) From bruce at leban.us Thu Oct 1 04:53:26 2015 From: bruce at leban.us (Bruce Leban) Date: Wed, 30 Sep 2015 19:53:26 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560C87A3.1020507@mrabarnett.plus.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> Message-ID: I don't understand the resistance to using ?? ?. ?[] ?() and the quest to contort some English words into this. The concepts "or" and "and" are natural language concepts and "if null" is not. The goal here should be readability and sometimes that readability is enhanced by being slightly different than other programming languages and sometimes it's For example, Java using -> for lambda is confusing to someone coming from C++ where -> means de-reference. C# avoided that confound by using =>. I'm sure someone thought that -> was more Javalike. Very few people writing/reading Python only use Python. There is a strong benefit to being consistent with other languages when there's not "one obvious" way to do it in "more Pythonic" syntax. So far, *none* of the alternatives are easier to read (for my taste) than the ? operators. As to the people that are questioning the need for any of this, I challenge you to look through your projects for code like this: Python: \bif\b.*\bis *(not *)?none *else C-like: ([=!]= *null\b|\bnull *[=!]=) *\? and see how many hits you find. If you want to avoid using None (or null) in your code, that's fine but the reality is that None is used extensively *and* there are *many* cases where we do exactly what these operators simplify. On Wed, Sep 30, 2015 at 6:08 PM, MRAB wrote: > >> It's only just occurred to me that there's a small inconsistency here. > The "?.", "?[" and "?(" will short-circuit on None, whereas the "??" > will short-circuit on non-None. > > Would that cause any confusion in practice? Personally, I don't think so and I never thought about this as an area of confusion. In the ?? case, it's obvious Two more notes: (1) If we have ?. ?() ?[] then spelling the ?? operator with a ? makes a lot of sense but mixing a word and an operator (like or?) just looks ugly to me. That said, if Guido declares absolutely no on ?. ?() ?[] and yes on ?? but it has to be spelled with words, my preference would be "*or else*". It doesn't really say "if is None" to me but (i) it's a syntax error right now; and (ii) the else in there implies some if-like logic is going on rather than just or-like logic: (foo or else bar) == (foo if foo is not None else bar). (2) As to ?= or ??=, I am less enamored of that but not because that case never occurs. I often find that the initialization code after if foo is null: is more than a single expression and therefore not amenable to ?=. That is, I don't just want to create an object and assign to the variable, I want to make initialization calls to it. --- Bruce Check out my new puzzle book: http://J.mp/ingToConclusions Get it free here: http://J.mp/ingToConclusionsFree (available on iOS) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Thu Oct 1 06:03:50 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 00:03:50 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> Message-ID: Akira Li <4kir4.1i at gmail.com> writes: >> 3. Cannot accept an iterator (goes through it twice, or permanently >> stores a reference to it, etc) > neither > > *iterable* is an object that you can pass to iter() to get *iterator*. > An iterable does not guarantee that it yields the same items twice. True or false?: It is reasonable to write algorithms that iterate twice over a passed-in iterable, with the expectation that said iterable will typically be an object (or a view of such an object) which will not be concurrently modified (e.g. by a different thread or by a side-effect of a callback) during the execution of the algorithm, but which does not behave in a useful way when given an iterator, a generator, or any other kind of iterable which exhibits similar behavior whereby the second and further attempts to iterate will yield no items. From emile at fenx.com Thu Oct 1 06:05:51 2015 From: emile at fenx.com (Emile van Sebille) Date: Wed, 30 Sep 2015 21:05:51 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <27D77A4D-7CDE-470D-BD55-AC25CE9F782B@yahoo.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <27D77A4D-7CDE-470D-BD55-AC25CE9F782B@yahoo.com> Message-ID: On 9/30/2015 7:24 PM, Andrew Barnert via Python-ideas wrote: > Also, the tutorial uses the phrases "data structures" or "data type" a few zillion times, apparently to avoid having to come up with a term that includes sequences, sets, dicts, and strings without being inaccurate. I've seen novices have no idea what "data structure" means, or get confused by what the difference between a "data type" and a "regular type" is. container? https://docs.python.org/3/library/collections.html Emile From random832 at fastmail.com Thu Oct 1 06:13:50 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 00:13:50 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <27D77A4D-7CDE-470D-BD55-AC25CE9F782B@yahoo.com> Message-ID: Andrew Barnert via Python-ideas writes: > I think the docs used to use the word "virtual" as a more specific > term than "lazy": a view onto an object that conceptually exists but > doesn't actually exist is "virtual" (like range, which is a view into > the infinite set of integers), but a view into a real object isn't > (like dict_keys, which has a reference to an actual dict), nor is > something that isn't conceptually view-like at all (like a RNG > iterator), even though they're all "lazy". If either "lazy" or "virtual" means that the contained objects don't exist as python objects until they are accessed, doesn't this extend to strings and arrays (and byte strings, byte arrays, and memory views)? From random832 at fastmail.com Thu Oct 1 06:19:08 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 00:19:08 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> Message-ID: Andrew Barnert via Python-ideas writes: > On Sep 30, 2015, at 18:08, MRAB wrote: >> >> It's only just occurred to me that there's a small inconsistency here. >> The "?.", "?[" and "?(" will short-circuit on None, whereas the "??" >> will short-circuit on non-None. >> >> Would that cause any confusion in practice? > > I noticed this when I was trying to write out grammar, sample ASTs, > and sample bytecode for these things. I went searching the thread and > saw no one had pointed it out. I went through docs and blogs for other > languages, and didn't see anyone pointing out, complaining about, or > offering to clear up any confusion. So I figured I wouldn't mention > it, and see if anyone else even noticed. How is it worse than the fact that and short-circuits on true whereas or short-circuits on false? Short-circuiting logically applies to the case that *can* be short-circuited. For the AST issue, I'm curious as to what you ended up doing about the whole-atom_expr nature of the short-circuiting and the fact that ASTs don't currently represent an atom_expr as a single object containing a list of subscript/attribute/call items? From 4kir4.1i at gmail.com Thu Oct 1 07:15:25 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 01 Oct 2015 08:15:25 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> Message-ID: <87bncj2mqa.fsf@gmail.com> Random832 writes: > Akira Li <4kir4.1i at gmail.com> writes: >>> 3. Cannot accept an iterator (goes through it twice, or permanently >>> stores a reference to it, etc) >> neither >> >> *iterable* is an object that you can pass to iter() to get *iterator*. >> An iterable does not guarantee that it yields the same items twice. > > True or false?: It is reasonable to write algorithms that iterate twice > over a passed-in iterable, with the expectation that said iterable will > typically be an object (or a view of such an object) which will not be > concurrently modified (e.g. by a different thread or by a side-effect of > a callback) during the execution of the algorithm, but which does not > behave in a useful way when given an iterator, a generator, or any other > kind of iterable which exhibits similar behavior whereby the second and > further attempts to iterate will yield no items. > True or false?: do all iterables return the same items twice? http://www.fallacyfiles.org/loadques.html Specific application may use more specific requirements e.g.: list(iterable): - does it mean that all iterables must be finite? - do we need a special word to describe what list() accepts? set(iterable): - does it mean that all iterables must yield hashable items? - do we need a special word to describe what set() accepts? dict(iterable): - does it mean that all iterables must yield pairs? - do we need a special word to describe what dict() accepts? You've got the idea: the word *iterable* may be used in the context when not all iterables are accepted. From rosuav at gmail.com Thu Oct 1 07:30:14 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Oct 2015 15:30:14 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151001013106.GC23642@ando.pearwood.info> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <20151001013106.GC23642@ando.pearwood.info> Message-ID: On Thu, Oct 1, 2015 at 11:31 AM, Steven D'Aprano wrote: > Personally, I don't like it, but using "n in range(a, b+1)" for testing > whether integer n falls within a particular range seems to be popular. I > don't know why they don't just write a <= n <= b, but it seems to be a > popular idiom for some weird reason. You could have a third arg to range, in which case it also adds in a modulo check. ChrisA From 4kir4.1i at gmail.com Thu Oct 1 07:39:51 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 01 Oct 2015 08:39:51 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> Message-ID: <87a8s32llk.fsf@gmail.com> Andrew Barnert via Python-ideas writes: > On Sep 30, 2015, at 19:04, Akira Li <4kir4.1i at gmail.com> wrote: >> >> Random832 writes: >> >>> Akira Li <4kir4.1i at gmail.com> writes: >>> >>>> Andrew Barnert via Python-ideas >>>> writes: >>>> ... >>>>> (The fact that we don't have a term for "non-iterator iterable", and >>>> >>>> All iterators are iterable but some iterables are not iterators. >>>> >>>> If your code accepts only iterators then use the term *iterator*. >>>> Otherwise the term *iterable* could be used. >>>> >>>> It is misleading to use *iterable* if your code only accepts iterators. >>>> >>>> If an iterable is an iterator; It is called *iterator*. The term >>>> *iterable* implies that some instances are not iterators. >>> >>> There are three (well, three and a half) kinds of code that consume >>> iterables, how would you describe each simply? >>> >>> 1. Does not call iter, simply calls next. Therefore cannot consume a >>> non-iterator iterable. >> iterator >> >>> 2. Calls iter, but can accept an iterator (e.g. only goes through it >>> once) >> iterable >> >>> 3. Cannot accept an iterator (goes through it twice, or permanently >>> stores a reference to it, etc) >> neither >> >> *iterable* is an object that you can pass to iter() to get *iterator*. >> An iterable does not guarantee that it yields the same items twice. > > And this is exactly the problem. We don't have any way to simply > describe this thing. Hence all the confusion in this thread, and in > similar discussions elsewhere, and even in the docs (e.g., describing > dict views as sequences and then going on to explain that they're not > actually sequences). Use *iterable* instead of "non-iterator iterable" -- it is that simple. "dict views" seems a pretty good term for dict views. Are you suggesting to call dict views "non-iterator iterable"? I don't see that it says more than that all dict views are iterable. It seems there is a bug in the glossary: the entry name should be "dict views", not just "view" that is too generic for the description. I've submitted a patch http://bugs.python.org/issue25286 > The fact that it took your previous message four paragraphs without > inventing a new term, to say what I said in one sentence with a new > term, demonstrates the problem. As does the fact that my attempted new > term, "non-iterator iterable", is sufficiently ugly and not > intuitively helpful enough that you felt the need to expand on it for > four paragraphs. I don't need 4 paragraphs to describe it: if you need an iterator; use the term *iterator* -- otherwise use *iterable* unless you need something more specific e.g., *seq* name is common for generic sequences I don't remember ever using "non-iterator iterable". "non-iterator iterable" does not qualify as more specific. You need to introduce new requirements to the type for that. From random832 at fastmail.com Thu Oct 1 07:45:11 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 01:45:11 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> Message-ID: Akira Li <4kir4.1i at gmail.com> writes: > I don't remember ever using "non-iterator iterable". "non-iterator > iterable" does not qualify as more specific. You need to introduce new > requirements to the type for that. The question is, how do you *simply* state the very common requirement for an iterable to not behave in a specific undesirable way that all iterators do, and that it is very uncommon for any iterable other than an iterator to do? Are you opposed to having a word for this concept at all, or do you just not like the terms other people are suggesting? From 4kir4.1i at gmail.com Thu Oct 1 07:59:24 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 01 Oct 2015 08:59:24 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> Message-ID: <87612r2koz.fsf@gmail.com> Random832 writes: > Akira Li <4kir4.1i at gmail.com> writes: >> I don't remember ever using "non-iterator iterable". "non-iterator >> iterable" does not qualify as more specific. You need to introduce new >> requirements to the type for that. > > The question is, how do you *simply* state the very common requirement > for an iterable to not behave in a specific undesirable way that all > iterators do, and that it is very uncommon for any iterable other than > an iterator to do? Are you opposed to having a word for this concept at > all, or do you just not like the terms other people are suggesting? That term is **iterable**. As I already said: Specific application may use more specific requirements e.g.: list(iterable): - does it mean that all iterables must be finite? - do we need a special word to describe what list() accepts? set(iterable): - does it mean that all iterables must yield hashable items? - do we need a special word to describe what set() accepts? dict(iterable): - does it mean that all iterables must yield pairs? - do we need a special word to describe what dict() accepts? You've got the idea: the word *iterable* may be used in the context when not all iterables are accepted. https://mail.python.org/pipermail/python-ideas/2015-October/036692.html From abarnert at yahoo.com Thu Oct 1 07:59:32 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 22:59:32 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> Message-ID: <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> On Sep 30, 2015, at 21:19, Random832 wrote: > > Andrew Barnert via Python-ideas > writes: > >>> On Sep 30, 2015, at 18:08, MRAB wrote: >>> >>> It's only just occurred to me that there's a small inconsistency here. >>> The "?.", "?[" and "?(" will short-circuit on None, whereas the "??" >>> will short-circuit on non-None. >>> >>> Would that cause any confusion in practice? >> >> I noticed this when I was trying to write out grammar, sample ASTs, >> and sample bytecode for these things. I went searching the thread and >> saw no one had pointed it out. I went through docs and blogs for other >> languages, and didn't see anyone pointing out, complaining about, or >> offering to clear up any confusion. So I figured I wouldn't mention >> it, and see if anyone else even noticed. > > How is it worse than the fact that and short-circuits on true whereas or > short-circuits on false? Why are you asking me how it's worse when my conclusion was that it's fine, in the sentence right after the part you quoted? > For the AST issue, I'm curious as to what you ended up doing about the > whole-atom_expr nature of the short-circuiting and the fact that ASTs > don't currently represent an atom_expr as a single object containing a > list of subscript/attribute/call items? I posted ?. examples earlier; I don't want to repeat the whole thing (especially after Guido pointed out that it probably wasn't helping anyone who didn't already get it). But briefly, the AST doesn't have to represent the short-circuiting here, any more than it does anywhere else that short-circuits; it just adds an uptalk flag to each Attribute node. At code generation time, any Attribute node that has uptalk=True has a JUMP_IF_NONE to after the primary-or-call (leaving the None attrib value on the stack) after the LOAD_ATTR. (Or, if you don't want to add a new bytecode, it has a string of three ops that do the equivalent.) The same works for ?[]. For ?(), I'm not sure what the desired semantics are, and for ?? it seemed obviously trivial so I didn't bother. From abarnert at yahoo.com Thu Oct 1 08:07:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 23:07:47 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <27D77A4D-7CDE-470D-BD55-AC25CE9F782B@yahoo.com> Message-ID: On Sep 30, 2015, at 21:13, Random832 wrote: > > Andrew Barnert via Python-ideas > writes: >> I think the docs used to use the word "virtual" as a more specific >> term than "lazy": a view onto an object that conceptually exists but >> doesn't actually exist is "virtual" (like range, which is a view into >> the infinite set of integers), but a view into a real object isn't >> (like dict_keys, which has a reference to an actual dict), nor is >> something that isn't conceptually view-like at all (like a RNG >> iterator), even though they're all "lazy". > > If either "lazy" or "virtual" means that the contained objects don't > exist as python objects until they are accessed, doesn't this extend to > strings and arrays (and byte strings, byte arrays, and memory views)? There's a sense in which that's true, and in some discussions (e.g., intimately involving the GC or object sharing or optimization of array algorithms) that would be the most relevant sense, but there's also a sense in which they concretely hold all the values in memory, and in most discussions (e.g., talking about generic sequence algorithms) that would be more relevant. I don't think there's a major problem here?we don't need to eliminate all ambiguity from our speech, only the ambiguity that actually gets in the way. From abarnert at yahoo.com Thu Oct 1 08:24:24 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 23:24:24 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <87a8s32llk.fsf@gmail.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> Message-ID: On Sep 30, 2015, at 22:39, Akira Li <4kir4.1i at gmail.com> wrote: > > Andrew Barnert via Python-ideas > writes: > >>> On Sep 30, 2015, at 19:04, Akira Li <4kir4.1i at gmail.com> wrote: >>> >>> Random832 writes: >>> >>>> Akira Li <4kir4.1i at gmail.com> writes: >>>> >>>>> Andrew Barnert via Python-ideas >>>>> writes: >>>>> ... >>>>>> (The fact that we don't have a term for "non-iterator iterable", and >>>>> >>>>> All iterators are iterable but some iterables are not iterators. >>>>> >>>>> If your code accepts only iterators then use the term *iterator*. >>>>> Otherwise the term *iterable* could be used. >>>>> >>>>> It is misleading to use *iterable* if your code only accepts iterators. >>>>> >>>>> If an iterable is an iterator; It is called *iterator*. The term >>>>> *iterable* implies that some instances are not iterators. >>>> >>>> There are three (well, three and a half) kinds of code that consume >>>> iterables, how would you describe each simply? >>>> >>>> 1. Does not call iter, simply calls next. Therefore cannot consume a >>>> non-iterator iterable. >>> iterator >>> >>>> 2. Calls iter, but can accept an iterator (e.g. only goes through it >>>> once) >>> iterable >>> >>>> 3. Cannot accept an iterator (goes through it twice, or permanently >>>> stores a reference to it, etc) >>> neither >>> >>> *iterable* is an object that you can pass to iter() to get *iterator*. >>> An iterable does not guarantee that it yields the same items twice. >> >> And this is exactly the problem. We don't have any way to simply >> describe this thing. Hence all the confusion in this thread, and in >> similar discussions elsewhere, and even in the docs (e.g., describing >> dict views as sequences and then going on to explain that they're not >> actually sequences). > > Use *iterable* instead of "non-iterator iterable" -- it is that simple. No it isn't. The word "iterable" just means "iterable". When you want to talk about sequences?a subtype of iterables?you don't just say "iterable", you say "sequence". And likewise, when you want to talk about iterables that aren't iterators, or iterables that are repeatable, or any other subtype of iterables, you have to use a word (or phrase) that actually means what you're saying. I don't know how to explain this any better. Everyone else seems to get it, but you just post the same reply to each of them that you posted to me when they try to explain further. What am I not getting across here? > "dict views" seems a pretty good term for dict views. > Are you suggesting to call dict views "non-iterator iterable"? Why would you think that? > I don't see that it says more than that all dict views are iterable. > > It seems there is a bug in the glossary: the entry name should be "dict > views", not just "view" that is too generic for the description. There's a much larger problem. The glossary says that dict views are sequences. They aren't. The actual documentation for dict views is a little better, because it explains that they're not actually sequences. But the problem is still there: what the docs are trying to say is that dict views are some kind of non-iterator iterable, but, because we don't have a term form that, they use the incorrect term "sequence". > I've submitted a patch http://bugs.python.org/issue25286 > >> The fact that it took your previous message four paragraphs without >> inventing a new term, to say what I said in one sentence with a new >> term, demonstrates the problem. As does the fact that my attempted new >> term, "non-iterator iterable", is sufficiently ugly and not >> intuitively helpful enough that you felt the need to expand on it for >> four paragraphs. > > I don't need 4 paragraphs to describe it: > > if you need an iterator; use the term *iterator* -- > otherwise use *iterable* unless you need something more specific e.g., > *seq* name is common for generic sequences Why would you use "seq" instead of "sequence" for the name of the abstract sequence type? And, more importantly, what name do you use when you need something more specific than "iterable", but less specific than "sequence"?as in the glossary entry for dict views, for example? > I don't remember ever using "non-iterator iterable". Why would you expect to remember using it, when you're replying to a message where I invented it for lack of an established better name (and in hopes that someone would come up with one)? > "non-iterator > iterable" does not qualify as more specific. You need to introduce new > requirements to the type for that. There are things that are iterables, that are not non-iterator iterables, but the reverse is not true. It's a name for a strict subset. Which means it's more specific. As for a new requirement: an iterable is a non-iterator iterable if its __iter__ method does not return self. (If you're going to argue that this requirement can't be checked by, e.g., a structural type checker, remember that neither is the distinction between sequence and mapping, and that doesn't mean they're the same type.) From abarnert at yahoo.com Thu Oct 1 08:34:34 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 23:34:34 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <87612r2koz.fsf@gmail.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> <87612r2koz.fsf@gmail.com> Message-ID: On Sep 30, 2015, at 22:59, Akira Li <4kir4.1i at gmail.com> wrote: > > Random832 writes: > >> Akira Li <4kir4.1i at gmail.com> writes: >>> I don't remember ever using "non-iterator iterable". "non-iterator >>> iterable" does not qualify as more specific. You need to introduce new >>> requirements to the type for that. >> >> The question is, how do you *simply* state the very common requirement >> for an iterable to not behave in a specific undesirable way that all >> iterators do, and that it is very uncommon for any iterable other than >> an iterator to do? Are you opposed to having a word for this concept at >> all, or do you just not like the terms other people are suggesting? > > That term is **iterable**. As I already said: > > Specific application may use more specific requirements e.g.: > > list(iterable): > > - does it mean that all iterables must be finite? > - do we need a special word to describe what list() accepts? > > set(iterable): > > - does it mean that all iterables must yield hashable items? > - do we need a special word to describe what set() accepts? > > dict(iterable): > > - does it mean that all iterables must yield pairs? > - do we need a special word to describe what dict() accepts? > > > You've got the idea: the word *iterable* may be used in the context when > not all iterables are accepted. > > https://mail.python.org/pipermail/python-ideas/2015-October/036692.html This is a link to a reply where you pasted exactly the same text as in this reply?and in a third one. What is that supposed to mean? I feel like you must be trying to get across something really important here, and it's my fault for not getting it, but I still can't get it. Can you try rewording it instead of just pasting the same text again and/or a link to the same text? If it helps, let me try to ask specific questions: Are you arguing one of the following: * there is no such thing as an iterable that isn't an iterator, or an iterable that is repeatable, or an iterable that provides a new iterator each time iter is called? * there are such things, but no corresponding property that can be used to characterize a set? * that such sets do exist, but are never useful to discuss? * that such sets may be useful to discuss, but the names I (and Terry and others) came up with are unhelpful? More concretely: the documentation for dict views goes out of its way to point out that these are not iterators, but a different kind of iterable that's more like a sequence (presumably meaning at least one of the three things above). But it does so inaccurately, by saying they are sequences, which is not true. How could it be rewritten to get that point across accurately, but still concisely and readably? From abarnert at yahoo.com Thu Oct 1 08:39:04 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 30 Sep 2015 23:39:04 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <27D77A4D-7CDE-470D-BD55-AC25CE9F782B@yahoo.com> Message-ID: <9F8724E9-9B86-4789-AD5C-491F6AC864F0@yahoo.com> On Sep 30, 2015, at 21:05, Emile van Sebille wrote: > >> On 9/30/2015 7:24 PM, Andrew Barnert via Python-ideas wrote: >> Also, the tutorial uses the phrases "data structures" or "data type" a few zillion times, apparently to avoid having to come up with a term that includes sequences, sets, dicts, and strings without being inaccurate. I've seen novices have no idea what "data structure" means, or get confused by what the difference between a "data type" and a "regular type" is. > > container? But that means something with a __contains__ test. Containers don't even have to be iterables. It's true that all of the types discussed in the tutorial are containers, but is that actually the meaning we're looking for, or just something that's coincidentally true? At any rate, even if that does work for the tutorial, I don't think it solves the more general problem. When I want to talk about iterables that give you a different, independent iterator each time you call __iter__, "container" is not the right word for that. Terry's "collection" seems like a better choice, because it doesn't already have a conflicting meaning. From brenbarn at brenbarn.net Thu Oct 1 08:44:04 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Wed, 30 Sep 2015 23:44:04 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> Message-ID: <560CD634.6080509@brenbarn.net> On 2015-09-30 23:24, Andrew Barnert via Python-ideas wrote: > There are things that are iterables, that are not non-iterator > iterables, but the reverse is not true. It's a name for a strict > subset. Which means it's more specific. > > As for a new requirement: an iterable is a non-iterator iterable if > its __iter__ method does not return self. Not sure I followed all the discussion of these terms, but is your main reason for wanting this term to describe the behavior that non-iterator iterables can be "restarted" (and are so restarted if reused in a different context)? Personally I prefer to take a duck-typing view and focus on what operations you can or can't do on these various things. Whether you call it a view or a virtual indexer or whatever is, to me, less important than what you can do with the object. I agree there are a number of relevant subcategories of objects here, some of which we have a name for and some that we don't. But I think it gets easier if we move from generic nouns like "view" to specific adjectives describing the behaviors the object support. Something like "re-entrant iterable" (meaning if you use it in two for loops right after each other you get the whole thing both times) would focus on that aspect of the behavior. Something like "random-accessible" or "sliceable" if we want to talk about iterables where we can "jump ahead" or slice if needed. It's an interesting idea to think about what kinds of operations (map, filter, etc.) could return iterables supporting what other kinds of operations. That is, can we make sure the result of map/filter can be sliced/indexed/reentered if the source can. To me the interesting question is which of these actual behaviors can usefully and non-mind-bendingly be preserved through map/filter/etc. manipulations. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From stephen at xemacs.org Thu Oct 1 09:13:06 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 1 Oct 2015 16:13:06 +0900 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: Message-ID: <22028.56578.901600.144640@turnbull.sk.tsukuba.ac.jp> Terry Reedy writes: > On 9/30/2015 1:28 PM, Chris Barker wrote: > > > But again, we could add indexing to enumerate, and have it do the ugly > > inefficient thing when it's using an underlying non-indexable iterator, > > If the ugly inefficient thing is to call list(iterable), then that does > not work with unbounded iterables. I think he means from itertools import islice a = list(islice(iterable, 0, 99))[42] > Or the input iterable might produce inputs at various times in the > future. Horrors! We'll have to add a "block=False" parameter to next(). (We can bikeshed on the default later.) Seriously, I think that one we just have to live with, just as we already live with it in any context where we access an iterable. Regards, From ncoghlan at gmail.com Thu Oct 1 09:59:11 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Oct 2015 17:59:11 +1000 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> Message-ID: On 1 October 2015 at 12:39, Andrew Barnert via Python-ideas wrote: > On Sep 30, 2015, at 18:08, MRAB wrote: >> >> It's only just occurred to me that there's a small inconsistency here. >> The "?.", "?[" and "?(" will short-circuit on None, whereas the "??" >> will short-circuit on non-None. >> >> Would that cause any confusion in practice? > > I noticed this when I was trying to write out grammar, sample ASTs, and sample bytecode for these things. I went searching the thread and saw no one had pointed it out. I went through docs and blogs for other languages, and didn't see anyone pointing out, complaining about, or offering to clear up any confusion. So I figured I wouldn't mention it, and see if anyone else even noticed. > > The fact that an experienced programmer like you didn't even notice it until after a zillion messages over a span of weeks, and apparently nobody else did at all, seems to imply that there's no dangerously misleading intuitive parallel here. > > (By the way, I have a feeling that including ?= will increase the risk of confusion, but I have no idea where that feeling comes from, so it may just be random noise in my head, maybe because I like ?. but don't particularly like ?= or something...) Because unlike "?.", "?[" and "?(", "?=" would also shortcircuit on "if not None" if expanded as a ternary expression: target = target if target is not None else default However, it's possible to make it consistent by instead expanding it as: if target is None: target = default As a result, one interesting way of looking at this problem is to ask: what if we *only* offered the conditional operations, *without* offering a binary null-coalescing operator? That is, we could allow conditional assignment: data ?= [] headers ?= {} arg ?= make_default() With the meaning: if data is None: data = [] if headers is None: headers = {} if arg is None: arg = make_default() That would deal directly with the idiomatic case of handling a default argument of "None" and replacing it with a mutable container or expensive to calculate default value. Multiple levels of coalescence would need to be spelled out as multiple statements rather than as chained binary operations (in this case, I think putting it all on one line helps make the "first non-None value wins" semantics clearer): title ?= user_title; title ?= local_default_title; title ?= global_default_title The semantics of dict.setdefault() could also potentially be made clearer, as "d[key] ?= make_default()" could become a short circuiting equivalent of "dict.setdefault(k, make_default())": try: d[key] except KeyError: d[key] = make_default() Custom sentinels would still need to be spelled out with an if statement: if arg is _sentinel: arg = make_default() If they were offered, conditional attribute access, conditional item lookup and conditional calls would use the same "LHS is None" check, but in a ternary expression rather than an if statement, with the following: title?.upper() person?['name'] func?(arg) being equivalent to: None if title is None else title.upper() None if person is None else person['name'] None if func is None else func(arg) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Thu Oct 1 17:10:28 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Oct 2015 01:10:28 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <87bncj2mqa.fsf@gmail.com> References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> Message-ID: <20151001151028.GD23642@ando.pearwood.info> On Thu, Oct 01, 2015 at 08:15:25AM +0300, Akira Li wrote: > True or false?: do all iterables return the same items twice? > http://www.fallacyfiles.org/loadques.html [Aside: I have no idea what point you are making with the above link.] Of course they don't necessarily do so, but those that don't are not necessarily well-behaved. In the case of sequences and collections, the concept is that (absent any explicit mutation operation), iterating over it twice *should* give the same results, that is the normal expectation. But that isn't enforced, we can write something that breaks that rule: class WeirdIterable: def __getitem__(self, i): if random.random() > 0.9: raise IndexError return random.choice(["fe", "fi", "fo", "fum"]) but most people would consider that to be a pathological case. Yes, you can do it, and maybe you have a reason to do so, but you can't expect other people's code to deal with it gracefully. In the case of iterators, the answer is *certainly not*. Iterators are designed for the express purpose of handling not just the "lazy sequence" case where you choose to calculate results on demand as an optimization, but the case where you *have no choice* because the results are coming from some source which may change from run to run, e.g. an external data source. An iterator *may* repeat if run twice, but there is no expectation that it will do so. It's not just that the rule about repeatability is not enforced, but that there is no such rule in the first place. (By the way, when I talk about running an iterator twice, I'm completely aware that technically you cannot ever do so. What I mean is to iterate over the object, then *recreate the object* in some sense, then iterate over it again.) > Specific application may use more specific requirements e.g.: > > list(iterable): > > - does it mean that all iterables must be finite? > - do we need a special word to describe what list() accepts? No, and no. In principle, list() will quite happily create an infinite list for you, if you have infinite memory :-) The fact that in practice lists are probably limited to something of the order of 2**64 items or less is a mere quality of implementation issue :-) But to be more serious, no, in context we should understand that lists have actual physical limits, and even finite iterables may not be capable of being turned into lists: def gen(): for i in range(10**10000): yield i Perfectly finite in size, but you cannot have a list that big. It's not just *infinite iterables* which are prohibited, that's just a special case of iterables that will provide more items than you have memory to store. And that's not a fixed limit, it will differ from machine to machine. [...] > You've got the idea: the word *iterable* may be used in the context when > not all iterables are accepted. Sure. But the distinction is that while there are a whole lot of different iterables: - iterables with a sufficiently small number of items - iterables of hashable items - iterables of (hashable key, item) pairs - iterables of prime numbers less than one million - iterables of strings containing exactly 1 vowel etc they are special cases and don't need specialised names. But there is a *general* distinction between two cases: - iterables which are iterators - iterables which are not iterators We have a name for the first set: "iterators". But we don't have a name for the second set. Andrew suggested "non-iterator iterables" is too clumsy for general use, and suggests we need a better name. You suggested "iterables", but that clearly cannot work, since iterators are a kind of iterable. -- Steve From christoph at grothesque.org Thu Oct 1 18:27:01 2015 From: christoph at grothesque.org (Christoph Groth) Date: Thu, 01 Oct 2015 18:27:01 +0200 Subject: [Python-ideas] asyncio: factoring-out of the general mechanism Message-ID: <877fn61rmy.fsf@grothesque.org> Hello, I have a question/idea about the support for asynchronous programming in Python. With Python 3.5, support for asynchronous programming has been added to the core Python language. The new language features, however, are only useful with an event loop, as provided (only?) by the asyncio stdlib module. I notice that already the class asyncio.AbstractEventLoop contains many things that to me seem very specific to particular applications. For examples that class contains placeholders for methods that expect IP host names and IP ports as arguments. In a world where networking is done in other ways than through IP, Python would be still useful. One could keep using most of the standard library that is not specific to IP networking. Asynchronous programming is a concept that is way more universal than many of the things in asyncio (IP ports, unix pipes, etc.). Has there been a discussion about separating the abstract bits needed for asynchronous programming in Python from the application-specific ones? Or would this be a bad idea for some reason? Christoph -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 810 bytes Desc: not available URL: From guido at python.org Thu Oct 1 19:07:43 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Oct 2015 10:07:43 -0700 Subject: [Python-ideas] asyncio: factoring-out of the general mechanism In-Reply-To: <877fn61rmy.fsf@grothesque.org> References: <877fn61rmy.fsf@grothesque.org> Message-ID: Well, what else did you have in mind? Remember that until we fix the GIL (which won't happen until Python 4) you won't get benefit from async programming unless you are overlapping with I/O, and (due to the limitations of select/poll/etc.) that is pretty much limited to network and IPC (disk I/O in particular cannot be overlapped in this way). The main other area of interest would be UI events (as in a browser or GUI) and for that I think an adaptation of the asyncio event loop will be suitable. (If you're not going to use them it's fine not to implement the network-specific methods on an event loop.) Note that the async/await keywords themselves have no direct link to an event loop -- they are just a different way to spell generators. It's just that they are only useful with asyncio or some other event loop library. On Thu, Oct 1, 2015 at 9:27 AM, Christoph Groth wrote: > Hello, > > I have a question/idea about the support for asynchronous programming in > Python. > > With Python 3.5, support for asynchronous programming has been added to > the core Python language. The new language features, however, are only > useful with an event loop, as provided (only?) by the asyncio stdlib > module. > > I notice that already the class asyncio.AbstractEventLoop contains many > things that to me seem very specific to particular applications. For > examples that class contains placeholders for methods that expect IP > host names and IP ports as arguments. > > In a world where networking is done in other ways than through IP, > Python would be still useful. One could keep using most of the standard > library that is not specific to IP networking. > > Asynchronous programming is a concept that is way more universal than > many of the things in asyncio (IP ports, unix pipes, etc.). Has there > been a discussion about separating the abstract bits needed for > asynchronous programming in Python from the application-specific ones? > Or would this be a bad idea for some reason? > > Christoph > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 1 19:12:38 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Oct 2015 03:12:38 +1000 Subject: [Python-ideas] secrets module -- secret.keeper? In-Reply-To: References: Message-ID: <20151001171238.GE23642@ando.pearwood.info> On Wed, Sep 30, 2015 at 01:55:49PM -0400, Jim J. Jewett wrote: > Will the secrets module offer any building blocks to actually protect a secret? > > e.g., > > an easy way to encrypt a file with a given password? > an encrypted datastore? I don't know. I think probably not -- they sound a bit too high level, at least for a first version. But if people want to propose functions to do so, they can be considered. > a getpass that works even in IDLE? I would expect that if getpass doesn't work in IDLE, that's a bug (in IDLE or getpass, I'm not sure :-) -- Steve From srkunze at mail.de Thu Oct 1 19:37:04 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 01 Oct 2015 19:37:04 +0200 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560B1E49.7050102@canterbury.ac.nz> <560BC0C3.4040502@btinternet.com> <560C68F0.4080600@mail.de> <560C95AF.1020501@btinternet.com> Message-ID: <560D6F40.8070802@mail.de> On 01.10.2015 04:30, Ryan Gonzalez wrote: > On September 30, 2015 9:08:47 PM CDT, Rob Cliffe wrote: >> On 30/09/2015 23:57, Sven R. Kunze wrote: >>> >>> The only reason why I would prefer "else" over the proposed >>> alternatives: it's already a reserved keyword and it's not really >>> necessary to waste another one. Otherwise, I don't care too much. >>> >>> Best, >>> Sven >> Are you suggesting that "a else b" is a possibility? >> That's a no-no because >> a if b else c else d >> is ambiguous; it could mean either of >> a if b else (c else d) >> (a if b else c) else d >> Rob >> > I already said that... And others already said "use parentheses" if in doubt. From srkunze at mail.de Thu Oct 1 20:15:12 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 01 Oct 2015 20:15:12 +0200 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> Message-ID: <560D7830.7030808@mail.de> On 01.10.2015 04:53, Bruce Leban wrote: > If you want to avoid using None (or null) in your code, that's fine > but the reality is that None is used extensively *and* there are > *many* cases where we do exactly what these operators simplify. Nothing what you said is wrong. However, it lacks the deeper understanding of the None issue. Just because everybody tells you the world is flat doesn't make it flat. Tell others it's round instead. Best, Sven From random832 at fastmail.com Thu Oct 1 20:16:22 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 14:16:22 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> Message-ID: <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> (Towards the bottom of this post I ask the question of *why* it's a bad thing for uptalk to "escape parentheses" - mentioning it up here so as to not bury the lead.) On Thu, Oct 1, 2015, at 01:59, Andrew Barnert wrote: > I posted ?. examples earlier; I don't want to repeat the whole thing > (especially after Guido pointed out that it probably wasn't helping > anyone who didn't already get it). But briefly, the AST doesn't have to > represent the short-circuiting here, any more than it does anywhere else > that short-circuits; it just adds an uptalk flag to each Attribute node. > At code generation time, any Attribute node that has uptalk=True has a > JUMP_IF_NONE to after the primary-or-call (leaving the None attrib value > on the stack) after the LOAD_ATTR. (Or, if you don't want to add a new > bytecode, it has a string of three ops that do the equivalent.) Four ops, actually - DUP_TOP LOAD_CONST(None) COMPARE_OP(is) POP_JUMP_IF_TRUE - unless I'm missing a more efficient way to do it. And it has to jump an arbitrary distance forward (however many calls, attributes, or subscripts are in the expression), not just "to after the one after". And the problem is, the AST can't differentiate (a.b).c from a.b.c, whereas (a?.b).c is *semantically different* from a?.b.c - I think a new AST structure is therefore necessary. At the very least you'd need something in the node corresponding to the uptalk to say _where_ to jump (i.e. how many levels to escape from). If designing an AST structure from scratch for such a language I think a?.b.c would absolutely not be represented as any kind of Y(X('a', 'b'), 'c') because it doesn't *make sense*. I was just wondering how you tackled this problem. > The same works for ?[]. For ?(), I'm not sure what the desired semantics > are The same as for the others, AIUI. >, and for ?? it seemed obviously trivial so I didn't bother. > My understanding of how the bytecode would work: a?.b.c.d?.e.f.g = LOAD_[whatever] a JUMP_IF_NONE * LOAD_ATTR b LOAD_ATTR c LOAD_ATTR d JUMP_IF_NONE * LOAD_ATTR e LOAD_ATTR f LOAD_ATTR g * this position is the target of both jumps (a?.b.c).d?.e.f.g LOAD_[whatever] a JUMP_IF_NONE * LOAD_ATTR b LOAD_ATTR c * target of first jump LOAD_ATTR d JUMP_IF_NONE ** LOAD_ATTR e LOAD_ATTR f LOAD_ATTR g ** target of second jump P.S. Now that I spell it out like that, it occurs to me that this second case is *not actually that useful*. You're guaranteeing an exception if a is None, which defeats the purpose of using uptalk on a?... at all. (It does change whether or not the side effects of a subscript/call arguments get evaluated, but it's not clear that this justifies the added complexity.) Maybe the uptalk should instead be apply to _any_ chain (by left operands) of AST Attribute/Subscript/Call nodes regardless of whether or not there are parentheses around them. It's been stated repeatedly that uptalk shouldn't "escape parentheses", but no-one's clearly stated *why* that should be the case. From chris.barker at noaa.gov Thu Oct 1 20:29:51 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 1 Oct 2015 11:29:51 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151001151028.GD23642@ando.pearwood.info> References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> Message-ID: On Thu, Oct 1, 2015 at 8:10 AM, Steven D'Aprano wrote: > But there > is a *general* distinction between two cases: > > - iterables which are iterators > - iterables which are not iterators > > We have a name for the first set: "iterators". But we don't have a name > for the second set. Andrew suggested "non-iterator iterables" is too > clumsy for general use, and suggests we need a better name. You > suggested "iterables", but that clearly cannot work, since iterators are > a kind of iterable. > sure -- but I've lost track of why it matters. "iterator" is well defined. And so is "iterable" -- why do we need to care whether the iterable returns itself when asked for an iterator? the term "sequence" is useful -- it defines certain behavior. So is the term "iterable", for the same reason. And it would be useful to say that given object is both a sequence and an iterable (are sequences iterable by definition?) But if why do you need to know that something is an iterable, but NOT an iterator? isn't that an implementation detail? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Thu Oct 1 20:33:26 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 14:33:26 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151001151028.GD23642@ando.pearwood.info> References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> Message-ID: <1443724406.464714.398903593.33D0C348@webmail.messagingengine.com> On Thu, Oct 1, 2015, at 11:10, Steven D'Aprano wrote: > On Thu, Oct 01, 2015 at 08:15:25AM +0300, Akira Li wrote: > > > True or false?: do all iterables return the same items twice? > > http://www.fallacyfiles.org/loadques.html > > [Aside: I have no idea what point you are making with the above link.] > > Of course they don't necessarily do so, but those that don't are not > necessarily well-behaved. I think what he is claiming, more or less, is that there is not a universal notion of "well-behaved" (this is true), or indeed *any* broadly-applicable notions of "well-behaved" (this is false). From python at mrabarnett.plus.com Thu Oct 1 20:36:20 2015 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 1 Oct 2015 19:36:20 +0100 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> Message-ID: <560D7D24.9030700@mrabarnett.plus.com> On 2015-10-01 19:16, Random832 wrote: > (Towards the bottom of this post I ask the question of *why* it's a bad > thing for uptalk to "escape parentheses" - mentioning it up here so as > to not bury the lead.) > > On Thu, Oct 1, 2015, at 01:59, Andrew Barnert wrote: >> I posted ?. examples earlier; I don't want to repeat the whole thing >> (especially after Guido pointed out that it probably wasn't helping >> anyone who didn't already get it). But briefly, the AST doesn't have to >> represent the short-circuiting here, any more than it does anywhere else >> that short-circuits; it just adds an uptalk flag to each Attribute node. >> At code generation time, any Attribute node that has uptalk=True has a >> JUMP_IF_NONE to after the primary-or-call (leaving the None attrib value >> on the stack) after the LOAD_ATTR. (Or, if you don't want to add a new >> bytecode, it has a string of three ops that do the equivalent.) > > Four ops, actually - DUP_TOP LOAD_CONST(None) COMPARE_OP(is) > POP_JUMP_IF_TRUE - unless I'm missing a more efficient way to do it. And > it has to jump an arbitrary distance forward (however many calls, > attributes, or subscripts are in the expression), not just "to after the > one after". > > And the problem is, the AST can't differentiate (a.b).c from a.b.c, > whereas (a?.b).c is *semantically different* from a?.b.c - I think a new > AST structure is therefore necessary. At the very least you'd need > something in the node corresponding to the uptalk to say _where_ to jump > (i.e. how many levels to escape from). > > If designing an AST structure from scratch for such a language I think > a?.b.c would absolutely not be represented as any kind of Y(X('a', 'b'), > 'c') because it doesn't *make sense*. I was just wondering how you > tackled this problem. > >> The same works for ?[]. For ?(), I'm not sure what the desired semantics >> are > > The same as for the others, AIUI. > >>, and for ?? it seemed obviously trivial so I didn't bother. >> > > My understanding of how the bytecode would work: > > a?.b.c.d?.e.f.g = > > LOAD_[whatever] a > JUMP_IF_NONE * > LOAD_ATTR b > LOAD_ATTR c > LOAD_ATTR d > JUMP_IF_NONE * > LOAD_ATTR e > LOAD_ATTR f > LOAD_ATTR g > * this position is the target of both jumps > > (a?.b.c).d?.e.f.g > > LOAD_[whatever] a > JUMP_IF_NONE * > LOAD_ATTR b > LOAD_ATTR c > * target of first jump > LOAD_ATTR d > JUMP_IF_NONE ** > LOAD_ATTR e > LOAD_ATTR f > LOAD_ATTR g > ** target of second jump > > P.S. > > Now that I spell it out like that, it occurs to me that this second case > is *not actually that useful*. You're guaranteeing an exception if a is > None, which defeats the purpose of using uptalk on a?... at all. > It's _not_ guaranteed that there'll be an exception if a is None; None does have _some_ attributes. > (It does change whether or not the side effects of a subscript/call > arguments get evaluated, but it's not clear that this justifies the > added complexity.) > > Maybe the uptalk should instead be apply to _any_ chain (by left > operands) of AST Attribute/Subscript/Call nodes regardless of whether or > not there are parentheses around them. It's been stated repeatedly that > uptalk shouldn't "escape parentheses", but no-one's clearly stated *why* > that should be the case. > 'and' and 'or' don't escape parentheses. I'd take it as a general rule that short-circuiting doesn't escape parentheses. From srkunze at mail.de Thu Oct 1 20:38:45 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 01 Oct 2015 20:38:45 +0200 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> Message-ID: <560D7DB5.1060802@mail.de> On 01.10.2015 20:29, Chris Barker wrote: > On Thu, Oct 1, 2015 at 8:10 AM, Steven D'Aprano > wrote: > > But there > is a *general* distinction between two cases: > > - iterables which are iterators > - iterables which are not iterators > > We have a name for the first set: "iterators". But we don't have a > name > for the second set. Andrew suggested "non-iterator iterables" is too > clumsy for general use, and suggests we need a better name. You > suggested "iterables", but that clearly cannot work, since > iterators are > a kind of iterable. > > > sure -- but I've lost track of why it matters. "iterator" is well > defined. And so is "iterable" -- why do we need to care whether the > iterable returns itself when asked for an iterator? > > the term "sequence" is useful -- it defines certain behavior. So is > the term "iterable", for the same reason. > > And it would be useful to say that given object is both a sequence and > an iterable (are sequences iterable by definition?) > > But if why do you need to know that something is an iterable, but NOT > an iterator? isn't that an implementation detail? You say some terms are useful because they define certain behavior. I don't question this, but what I find questionable is the proliferation of all this equally-sounding and -feeling concepts. Your reaction supports that observation. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Thu Oct 1 20:41:15 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 14:41:15 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> Message-ID: <1443724875.466486.398907305.46E7BFE5@webmail.messagingengine.com> On Thu, Oct 1, 2015, at 14:29, Chris Barker wrote: > But if why do you need to know that something is an iterable, but NOT an > iterator? isn't that an implementation detail? Because an iterator *cannot possibly* allow you to loop through the contents twice [either one after the other or in parallel], whereas *most* non-iterator iterables do allow this. This (among other things such as representing a well-defined finite bag of values) is the property we're really chasing, "non-iterator iterable" is just a clumsy and inaccurate way of saying it. (I'm actually moderately disappointed, incidentally, that there's no easy way to create e.g. an iterable that will spin up a fresh copy of the same generator each time it's called. But it's easy enough to make a decorator for that.) From random832 at fastmail.com Thu Oct 1 20:50:07 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 14:50:07 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560D7D24.9030700@mrabarnett.plus.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> <560D7D24.9030700@mrabarnett.plus.com> Message-ID: <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> On Thu, Oct 1, 2015, at 14:36, MRAB wrote: > It's _not_ guaranteed that there'll be an exception if a is None; None > does have _some_ attributes. Well, it hasn't got a "d" attribute, but point taken. However, wanting to build expressions like this that will sometimes operate on the attributes of None and other times operate on the attributes of the non-None object normally expected to be present seems like an *extremely* obscure thing to want. > 'and' and 'or' don't escape parentheses. I'd take it as a general rule > that short-circuiting doesn't escape parentheses. Sure they do. In what way is (a and b) and c different from a and b and c? Heck, (a and b) and (c and d) even compiles to the same bytecode as other groupings [demonstrating it has the same actual semantics] *even though it is a different AST*. But even if it didn't, it would just be jumping to another jump opcode. It doesn't escape being mixed with a different kind of expression, which is sometimes accomplished with parentheses, but that's not really the same thing. From steve at pearwood.info Thu Oct 1 21:37:09 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Oct 2015 05:37:09 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> Message-ID: <20151001193709.GF23642@ando.pearwood.info> On Thu, Oct 01, 2015 at 11:29:51AM -0700, Chris Barker wrote: > On Thu, Oct 1, 2015 at 8:10 AM, Steven D'Aprano wrote: > > > But there > > is a *general* distinction between two cases: > > > > - iterables which are iterators > > - iterables which are not iterators > > > > We have a name for the first set: "iterators". But we don't have a name > > for the second set. Andrew suggested "non-iterator iterables" is too > > clumsy for general use, and suggests we need a better name. You > > suggested "iterables", but that clearly cannot work, since iterators are > > a kind of iterable. > > > > sure -- but I've lost track of why it matters. "iterator" is well defined. > And so is "iterable" -- why do we need to care whether the iterable returns > itself when asked for an iterator? In and of itself, it probably isn't, except as a short-cut for deciding whether something is an iterator. [...] > But if why do you need to know that something is an iterable, but NOT an > iterator? isn't that an implementation detail? I forget the original context -- I think it was Andrew who first mentioned this. Possibly over confusion about (x)range. But in general, it's important because: - iterators are not random access, other iterables typically are; - iterators are one-shot (cannot be restarted), other iterables are typically re-runnable. This makes a difference. Just a few days ago, somebody mis-reported a supposed "bug" in all() and any(). For example: values = (x%5 == 3 for x in range(8)) print(list(values)) print(all(values)) # should return False Obvious error is obvious: having printed out the values from the generator expression, values is now exhausted, and all() of the empty set is True (vacuous truth). The difference between general iterables which may or may not be one-shot iterators, and those which are definitely not iterators, is not always just an implementation detail. -- Steve From steve at pearwood.info Thu Oct 1 21:48:43 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Oct 2015 05:48:43 +1000 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> References: <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> <560D7D24.9030700@mrabarnett.plus.com> <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> Message-ID: <20151001194843.GG23642@ando.pearwood.info> On Thu, Oct 01, 2015 at 02:50:07PM -0400, Random832 wrote: > On Thu, Oct 1, 2015, at 14:36, MRAB wrote: [...] > > 'and' and 'or' don't escape parentheses. I'd take it as a general rule > > that short-circuiting doesn't escape parentheses. > > Sure they do. In what way is (a and b) and c different from a and b and > c? Heck, (a and b) and (c and d) even compiles to the same bytecode as > other groupings [demonstrating it has the same actual semantics] *even > though it is a different AST*. But even if it didn't, it would just be > jumping to another jump opcode. I don't see what this has to do with null aware attribute access. If I wrote: (spam?.eggs.cheese).__class__ the obvious intention is that it should evaluate like: temp = spam?.eggs.cheese temp.__class__ only without the use of a temporary name. -- Steve From python at mrabarnett.plus.com Thu Oct 1 22:07:26 2015 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 1 Oct 2015 21:07:26 +0100 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> References: <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> <560D7D24.9030700@mrabarnett.plus.com> <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> Message-ID: <560D927E.5030800@mrabarnett.plus.com> On 2015-10-01 19:50, Random832 wrote: > On Thu, Oct 1, 2015, at 14:36, MRAB wrote: >> It's _not_ guaranteed that there'll be an exception if a is None; None >> does have _some_ attributes. > > Well, it hasn't got a "d" attribute, but point taken. However, wanting > to build expressions like this that will sometimes operate on the > attributes of None and other times operate on the attributes of the > non-None object normally expected to be present seems like an > *extremely* obscure thing to want. > >> 'and' and 'or' don't escape parentheses. I'd take it as a general rule >> that short-circuiting doesn't escape parentheses. > > Sure they do. In what way is (a and b) and c different from a and b and > c? Heck, (a and b) and (c and d) even compiles to the same bytecode as > other groupings [demonstrating it has the same actual semantics] *even > though it is a different AST*. But even if it didn't, it would just be > jumping to another jump opcode. > If a is falsy, it short-circuits "a and b". The parenthesised expression returns a falsy result. That falsy result then short-circuits "(a and b) and c". It happens to show the same behaviour as "a and b and c" and can be optimised to that. Well, that's my opinion, anyway! > It doesn't escape being mixed with a different kind of expression, which > is sometimes accomplished with parentheses, but that's not really the > same thing. > From p.f.moore at gmail.com Thu Oct 1 22:11:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Oct 2015 21:11:58 +0100 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <1443724875.466486.398907305.46E7BFE5@webmail.messagingengine.com> References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <1443724875.466486.398907305.46E7BFE5@webmail.messagingengine.com> Message-ID: On 1 October 2015 at 19:41, Random832 wrote: > On Thu, Oct 1, 2015, at 14:29, Chris Barker wrote: >> But if why do you need to know that something is an iterable, but NOT an >> iterator? isn't that an implementation detail? > > Because an iterator *cannot possibly* allow you to loop through the > contents twice [either one after the other or in parallel], whereas > *most* non-iterator iterables do allow this. This (among other things > such as representing a well-defined finite bag of values) is the > property we're really chasing, "non-iterator iterable" is just a clumsy > and inaccurate way of saying it. If I understand what you mean by "non-iterator iterable", then a long time ago, there was a similar discussion and the term "reiterable" was used (Google will probably find references). Nothing ever came of the discussion - if I recall, there was a lot of theoretical debate, but few practical use cases. Anyone wanting to avoid a long, inconclusive discussion should probably chase up that old thread and see if anything new has been added this time around :-) Paul From random832 at fastmail.com Thu Oct 1 22:36:53 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 16:36:53 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560D927E.5030800@mrabarnett.plus.com> References: <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> <560D7D24.9030700@mrabarnett.plus.com> <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> <560D927E.5030800@mrabarnett.plus.com> Message-ID: <1443731813.493510.399004009.2A460957@webmail.messagingengine.com> On Thu, Oct 1, 2015, at 16:07, MRAB wrote: > If a is falsy, it short-circuits "a and b". > > The parenthesised expression returns a falsy result. > > That falsy result then short-circuits "(a and b) and c". > > It happens to show the same behaviour as "a and b and c" and can be > optimised to that. Er... that's what a and b and c *is*. The 'and' operator is a left-associative binary operator. ((a and b) and c) is literally the same AST as (a and b and c). Being optimized to "short-circuit the whole thing" is an optimization for both of them. The naive way you describe of evaluating each one in turn is also the same for both of them. They are in fact both being optimized to (a and (b and c)) [rather, to the same byte code that a 'naive' implementation would still generate for that expression]. From random832 at fastmail.com Thu Oct 1 22:37:28 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 16:37:28 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <20151001194843.GG23642@ando.pearwood.info> References: <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> <560D7D24.9030700@mrabarnett.plus.com> <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> <20151001194843.GG23642@ando.pearwood.info> Message-ID: <1443731848.493597.398999825.4FD7CC16@webmail.messagingengine.com> On Thu, Oct 1, 2015, at 15:48, Steven D'Aprano wrote: > I don't see what this has to do with null aware attribute access. > > If I wrote: > > (spam?.eggs.cheese).__class__ > > the obvious intention is that it should evaluate like: > > temp = spam?.eggs.cheese > temp.__class__ Why would you need to do that, though? What's the practical use case for this, that you need to do it frequently enough to want a way to do it without using a temporary? Or do you think that, as the thing that it short-circuits to the end of, "same atom_expr unbroken by parentheses" [a concept which, I remind you, *does not exist* in AST] is somehow simpler or easier to explain than "same chain of Attribute/Call/Subscript operations"? From ron3200 at gmail.com Thu Oct 1 22:57:18 2015 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 1 Oct 2015 15:57:18 -0500 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> Message-ID: On 10/01/2015 02:59 AM, Nick Coghlan wrote: > As a result, one interesting way of looking at this problem is to > ask: what if we *only* offered the conditional operations, *without* > offering a binary null-coalescing operator? How languages treat None, null, and nil, are very fundamental aspects of how they work. For example I noticed that if we add an None specific 'or' operator, we may want a None specific 'and' operator too. And in order to use those in conditional statements, it makes sense to have None specific 'if' and 'while'. So it may be a slippery slope. (Or they may be good to have. I'm very undecided still.) > That is, we could allow conditional assignment: > > data ?= [] headers ?= {} arg ?= make_default() > > With the meaning: > > if data is None: data = [] if headers is None: headers = {} if arg is > None: arg = make_default() Is the above three statements, three expressions, or one long expression? I was contemplating the possibility of having a variation of "is" as a way to test for not-null, but it leads down the same path. Once you have one None specific bool operator, the rest are needed to make full use of it. An Observation... In many discussions here, the same shortcut problem keeps coming up when ever we don't want to evaluate a functions arguments too early. So maybe what we really need is a syntax for a conditional partial where some arguments aren't evaluated until asked for. The ?( is almost that, but some way to short cut the arguments would still be needed. One possibility is to have them all as optional only, and just not ask for them. And to go out on a limb... ;-) Another possibility is to have a *special magic callable* that when called skips the argument evaluation and returns None. NoneCall(these, args, are, never, evaluated) In this case the Magic is underwhelming as it does absolutely nothing except return None. I think I can live with that. Then a special method on objects, obj __cond__call__() can return a NoneCall to skip the arguments on the right side of ?(, or return another callable to use them. This is a bit like decorators in that to use the arguments you need to return a callable. So I think maybe the @ symbol would be better. I can see this being used in decorators to apply decorators conditionally. The case of a Null default might be done by just having a conditional call on None, unpack the arguments. So it's __cond_call__ method if it had one, would be an identity function. value = obj@(default) # if obj is None: value = default # if obj has __cond_call__ method # it calls it with the arguments. # else TypeError So the '@(', or '?(' if preferred, may be the only one we need. The rest might be implementable with it by defining __cond_call__ methods on the objects. Cheers, Ron From jbvsmo at gmail.com Thu Oct 1 23:00:44 2015 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Thu, 1 Oct 2015 18:00:44 -0300 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <560D927E.5030800@mrabarnett.plus.com> References: <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> <560D7D24.9030700@mrabarnett.plus.com> <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> <560D927E.5030800@mrabarnett.plus.com> Message-ID: I've been thinking of practical uses for this new None aware operator and although I really like the idea, I believe this may become too weird to use if some things are not taken care of. Are all of those things meant to work? (or am I just crazy?) -- assuming C#-like syntax: for el in re.match(foo, bar)?.groups() ?? []: print(el.upper()) ------ # Assuming this function for readability def is_useful(x): return x is not None if not foo?: # Should read like "not is_useful(foo)" print('foo is None') # With this syntax it is easier to correct all those pieces of code "if not x:" where it should be "if x is None:" if foo??: # A shorter version possibly meaning "foo ?? True" or "foo is None" print('foo is still None') ----- # Are operators other than [] and () allowed? bar = foo? + 1 # For "foo?.__add__(1)" it would work, but for the realistic # "type(foo)?.__add__(foo, 1)" it does not translate correctly bar = foo ?+ 1 # Another spelling. Is this any better?? bar = foo? / baz? / 2 # How would this work? '(foo / baz) / 2' or 'foo / (bar / 2)' ----- Also for correctness, the shouldn't the '?=' operator be '??=' -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Thu Oct 1 23:12:26 2015 From: random832 at fastmail.com (Random832) Date: Thu, 01 Oct 2015 17:12:26 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> Message-ID: <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: > And to go out on a limb... ;-) > > Another possibility is to have a *special magic callable* that when > called skips the argument evaluation and returns None. That's dangerous talk indeed. Special magic callables are Lisp territory. ;) And I don't even know how you'd implement it efficiently without them being known at compile-time. I guess at *every* callsite you could test if the callable is magic, and if it is evaluate the arguments, and if it's not just pass in a lambda that will return the arguments. But you've got to generate those lambdas for *all* callsites, even the vast majority that will never be a magic callable. What if the callable only wants *some* of the arguments? Hey, if this had existed back then the ternary operator could have been a normal function - instead of (b() if a else c()) just do iif(a, b(), c()). And is this going to be fully general? I.e. should this be supported for regular operators? If __add__ is magic does + do this, for example? From abarnert at yahoo.com Thu Oct 1 23:45:22 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 1 Oct 2015 14:45:22 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <063100D7-C750-4318-BE76-29C6D14C446A@yahoo.com> <1443723382.460965.398871433.5A10F489@webmail.messagingengine.com> <560D7D24.9030700@mrabarnett.plus.com> <1443725407.467972.398914841.50CDD4B3@webmail.messagingengine.com> <560D927E.5030800@mrabarnett.plus.com> Message-ID: On Oct 1, 2015, at 14:00, Jo?o Bernardo wrote: > > I've been thinking of practical uses for this new None aware operator and although I really like the idea, I believe this may become too weird to use if some things are not taken care of. > Are all of those things meant to work? (or am I just crazy?) -- assuming C#-like syntax: > > > for el in re.match(foo, bar)?.groups() ?? []: > print(el.upper()) Yes. > > ------ > > # Assuming this function for readability > def is_useful(x): > return x is not None > > if not foo?: # Should read like "not is_useful(foo)" > print('foo is None') > # With this syntax it is easier to correct all those pieces of code "if not x:" where it should be "if x is None:" No. Neither this, nor any of your other examples below, are meant to work. The proposal adds a binary operator ??, three primary operations ?., ?[], and ?(), and possibly a new assignment ?=. It does not add a unary postfix operator ?, and you can't infer one from any of those forms any more than you can infer a unary postfix + from +=. While there was some discussion very early in the thread about whether a unary postfix ? operator could be used instead of all these separate things, but it was quickly realized that it doesn't express short-circuiting right for null-conditional access, doesn't make any sense at all for binary null coalescing, and does the wrong thing for null-conditional assignment, so Guido soundly rejected it and nobody disagreed. Guido also raised the question of whether we should uptalk most operators, skipping only the ones that make no sense, or only update a few that have compelling use cases. People had a bit of fun exploring the former, but I think everyone agreed that the latter makes a lot more sense, and that's what's in the preliminary PEP. So, under the proposal under discussions, this example, and your other examples, are all syntax errors, just like "if foo+:" and "if foo++:" are syntax errors. > if foo??: # A shorter version possibly meaning "foo ?? True" or "foo is None" > print('foo is still None') > > ----- > # Are operators other than [] and () allowed? > bar = foo? + 1 # For "foo?.__add__(1)" it would work, but for the realistic > # "type(foo)?.__add__(foo, 1)" it does not translate correctly > bar = foo ?+ 1 # Another spelling. Is this any better?? > > bar = foo? / baz? / 2 # How would this work? '(foo / baz) / 2' or 'foo / (bar / 2)' > > ----- > > Also for correctness, the shouldn't the '?=' operator be '??=' > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Oct 2 00:11:23 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 1 Oct 2015 15:11:23 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> Message-ID: <3865BD76-6CCB-4920-81E8-F0001D3B7A8D@yahoo.com> On Oct 1, 2015, at 14:12, Random832 wrote: > >> On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: >> And to go out on a limb... ;-) >> >> Another possibility is to have a *special magic callable* that when >> called skips the argument evaluation and returns None. > > That's dangerous talk indeed. Special magic callables are Lisp > territory. ;) Despite the smiley at the end, I think that second sentence is exactly the issue here. This is basically fexprs, with the same pros and cons. The language doesn't have to provide new short-circuiting operators like if-else or ?? or other kinds of flow-control expressions because the end user can write them himself easily. On the other hand, because the end user can write flow-control expressions himself, every project ends up being written in a similar but semantically different language. (Of course hardcore Lisp advocates claim that con is also a pro: it means you can spend the first 80% of a project building up a language that makes the application trivial, and the last 20% coding and debugging the application. Sure, anyone who wasn't involved in that first 80% is never going to be able to understand the code you wrote in the last 20%, but really, who else in the world is smart enough to grok your brilliant code anyway, so where's the problem? Fortunately, no one has outlawed Lisp, so those people can continue to use it and don't have to use horribly restrictive languages like Python or Scala like all of us dumb sheeple, so we can ignore that.) > And I don't even know how you'd implement it efficiently without them > being known at compile-time. > > I guess at *every* callsite you could test if the callable is magic, and > if it is evaluate the arguments, and if it's not just pass in a lambda > that will return the arguments. But you've got to generate those lambdas > for *all* callsites, even the vast majority that will never be a magic > callable. > > What if the callable only wants *some* of the arguments? Hey, if this > had existed back then the ternary operator could have been a normal > function - instead of (b() if a else c()) just do iif(a, b(), c()). > > And is this going to be fully general? I.e. should this be supported for > regular operators? If __add__ is magic does + do this, for example? From chris.barker at noaa.gov Fri Oct 2 01:08:54 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 1 Oct 2015 16:08:54 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <1443724875.466486.398907305.46E7BFE5@webmail.messagingengine.com> References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <1443724875.466486.398907305.46E7BFE5@webmail.messagingengine.com> Message-ID: On Thu, Oct 1, 2015 at 11:41 AM, Random832 wrote: > > But if why do you need to know that something is an iterable, but NOT an > > iterator? isn't that an implementation detail? > > Because an iterator *cannot possibly* allow you to loop through the > contents twice [either one after the other or in parallel], whereas > *most* non-iterator iterables do allow this. This (among other things > such as representing a well-defined finite bag of values) is the > property we're really chasing, "non-iterator iterable" is just a clumsy > and inaccurate way of saying it. > um, then shod;nt you simply describe the iterator as an iterator? so any "iterable" would be assumed to be a non-iterator iterable. I guess this all comes about because we don't want to have to write this: for i in iter(an_iterable): ..... i.e have a different interface for interable and an iterable but I'm still lost on when tha all has to be spelled out... And back the original question, for enumerate: OK, you can't really have it e both an iterator AND a sequence, but couldn't it be an iterator and support indexing? Though I'm starting to wonder about the use case: enumerate() is a way to get the items in an iterable and an index at the same time -- so if you want to pass in a sequence, and index the result, why not just index into the sequence in the first place?? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 2 01:13:37 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 1 Oct 2015 16:13:37 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151001193709.GF23642@ando.pearwood.info> References: <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <20151001193709.GF23642@ando.pearwood.info> Message-ID: On Thu, Oct 1, 2015 at 12:37 PM, Steven D'Aprano wrote: > This makes a difference. Just a few days ago, somebody mis-reported a > supposed "bug" in all() and any(). For example: > > values = (x%5 == 3 for x in range(8)) > print(list(values)) > print(all(values)) # should return False > > Obvious error is obvious: having printed out the values from the > generator expression, values is now exhausted, and all() of the empty > set is True (vacuous truth). The difference between general iterables > which may or may not be one-shot iterators, and those which are > definitely not iterators, is not always just an implementation detail. they used a generator expression, when they clearly wanted a list comprehension -- so yes, it matters what they were getting, I don't know that adding more vocabulary would help prevent people from making that mistake... if they had been smart enough to call the list() again, before claiming there was a bug in all -- it may have explained itself. -Chris > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From 4kir4.1i at gmail.com Fri Oct 2 02:26:31 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Fri, 02 Oct 2015 03:26:31 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> <87612r2koz.fsf@gmail.com> Message-ID: <87y4fm15fs.fsf@gmail.com> Andrew Barnert via Python-ideas writes: > On Sep 30, 2015, at 22:59, Akira Li <4kir4.1i at gmail.com> wrote: ... >> You've got the idea: the word *iterable* may be used in the context when >> not all iterables are accepted. >> >> https://mail.python.org/pipermail/python-ideas/2015-October/036692.html > > This is a link to a reply where you pasted exactly the same text as in > this reply?and in a third one. What is that supposed to mean? It means exactly what it says -- literally: "the word *iterable* may be used in the context when not all iterables are accepted." and list(iterable), set(iterable), dict(iterable) are the specific examples. It is a statement of "how it *is*" and that it is acceptable in my view and there is no need to change it. Obviously, list/set/dict docs describe what subset of iterables they accept. If you agree on that then there is no disagreement. And it should answer the questions from your post. If you disagree then to ground the discussion what _specific_ places in the documentation would you like to change? ... > More concretely: the documentation for dict views goes out of its way > to point out that these are not iterators, but a different kind of > iterable that's more like a sequence (presumably meaning at least one > of the three things above). But it does so inaccurately, by saying > they are sequences, which is not true. How could it be rewritten to > get that point across accurately, but still concisely and readably? "How could it be rewritten": I remember posting the link to Python issue already http://bugs.python.org/issue25286 From tjreedy at udel.edu Fri Oct 2 02:46:54 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 1 Oct 2015 20:46:54 -0400 Subject: [Python-ideas] secrets module -- secret.keeper? In-Reply-To: <20151001171238.GE23642@ando.pearwood.info> References: <20151001171238.GE23642@ando.pearwood.info> Message-ID: On 10/1/2015 1:12 PM, Steven D'Aprano wrote: > On Wed, Sep 30, 2015 at 01:55:49PM -0400, Jim J. Jewett wrote: > >> Will the secrets module offer any ... >> a getpass that works even in IDLE? > I would expect that if getpass doesn't work in IDLE, that's a bug (in > IDLE or getpass, I'm not sure :-) Since Jim did not explain the implied 'does not work' I will guess that 'works' means replacing all typed or pasted chars with '*' before they are echoed to the screen. One can do this with tk(inter) and, I presume, any or all gui frameworks in use with Python. Text mode is tricker as system-specific commands are needed, and they may not work when tk (or another gui framework?) has control of the keyboard and screen. -- Terry Jan Reedy From njs at pobox.com Fri Oct 2 02:52:34 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 1 Oct 2015 17:52:34 -0700 Subject: [Python-ideas] secrets module -- secret.keeper? In-Reply-To: References: Message-ID: On Wed, Sep 30, 2015 at 10:55 AM, Jim J. Jewett wrote: > Will the secrets module offer any building blocks to actually protect a secret? > > e.g., > > an easy way to encrypt a file with a given password? > an encrypted datastore? -100 These are *waayyyyy* too complicated and subtle problems to try to handle in the stdlib. We cannot possibly do an acceptably good job here. > a getpass that works even in IDLE? This sounds like a bug in getpass and/or IDLE that should just be fixed. -n -- Nathaniel J. Smith -- http://vorpus.org From tjreedy at udel.edu Fri Oct 2 02:55:51 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 1 Oct 2015 20:55:51 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> Message-ID: On 9/30/2015 10:31 PM, Andrew Barnert via Python-ideas wrote: > On Sep 30, 2015, at 19:04, Akira Li <4kir4.1i at gmail.com> wrote: >> >> Random832 writes: >> >>> Akira Li <4kir4.1i at gmail.com> writes: >>> >>>> Andrew Barnert via Python-ideas >>>> writes: >>>> ... >>>>> (The fact that we don't have a term for "non-iterator iterable", and >>>> >>>> All iterators are iterable but some iterables are not iterators. >>>> >>>> If your code accepts only iterators then use the term *iterator*. >>>> Otherwise the term *iterable* could be used. >>>> >>>> It is misleading to use *iterable* if your code only accepts iterators. >>>> >>>> If an iterable is an iterator; It is called *iterator*. The term >>>> *iterable* implies that some instances are not iterators. >>> >>> There are three (well, three and a half) kinds of code that consume >>> iterables, how would you describe each simply? >>> >>> 1. Does not call iter, simply calls next. Therefore cannot consume a >>> non-iterator iterable. >> iterator >> >>> 2. Calls iter, but can accept an iterator (e.g. only goes through it >>> once) >> iterable >> >>> 3. Cannot accept an iterator (goes through it twice, or permanently >>> stores a reference to it, etc) >> neither re-iterable (with implication of same sequence of yields) I have used this for years on python list and do not think I am unique. >> *iterable* is an object that you can pass to iter() to get *iterator*. >> An iterable does not guarantee that it yields the same items twice. > > We don't have any way to simply describe this thing. Depends of the 'we'. -- Terry Jan Reedy From 4kir4.1i at gmail.com Fri Oct 2 03:47:45 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Fri, 02 Oct 2015 04:47:45 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <1443724406.464714.398903593.33D0C348@webmail.messagingengine.com> Message-ID: <87lhbm11oe.fsf@gmail.com> Random832 writes: > On Thu, Oct 1, 2015, at 11:10, Steven D'Aprano wrote: >> On Thu, Oct 01, 2015 at 08:15:25AM +0300, Akira Li wrote: >> ... >> Of course they don't necessarily do so, but those that don't are not >> necessarily well-behaved. > > I think what he is claiming, more or less, is that there is not a > universal notion of "well-behaved" (this is true), or indeed *any* > broadly-applicable notions of "well-behaved" (this is false). I would say that "well-behaved" "non-iterator iterable" is a strict subset of "non-iterator iterable". You probably want "reiterable" word that I see mentioned in the thread. I don't know whether *reiterable* implies that it produces the same items the second time but it certainly implies that next(iter(reiterable)) _may produce something if_ list(reiterable) call is successful. Perhaps *rerunnable* (that I also see mentioned in the thread) more strongly implies that the same items should be produced. From 4kir4.1i at gmail.com Fri Oct 2 03:59:15 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Fri, 02 Oct 2015 04:59:15 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> Message-ID: <87k2r61158.fsf@gmail.com> Andrew Barnert via Python-ideas writes: > On Sep 30, 2015, at 22:39, Akira Li <4kir4.1i at gmail.com> wrote: ... >> Use *iterable* instead of "non-iterator iterable" -- it is that simple. > > No it isn't. The word "iterable" just means "iterable". When you want > to talk about sequences?a subtype of iterables?you don't just say > "iterable", you say "sequence"... "iterables that aren't iterators" unlike sequences do not introduce new requirements -- __iter__ returning non-self doesn't count -- the return value is still an _arbitrary_ iterator e.g., it may return the same iterator each time. Sequences on the other hand do introduce new requirements (__len__, __getitem__ and their specific semantics that distinguishes them from Mappings). > ... And likewise, when you want to talk about iterables that aren't > iterators, or iterables that are repeatable, or any other subtype of > iterables, you have to use a word (or phrase) that actually means what > you're saying. I have no objection to the phrase "repeatable iterable" because it does introduce a useful distinction. ... >> if you need an iterator; use the term *iterator* -- >> otherwise use *iterable* unless you need something more specific e.g., >> *seq* name is common for generic sequences > > Why would you use "seq" instead of "sequence" for the name of the abstract sequence type? I meant in the code e.g., random.choice(seq), to communicate that an arbitrary iterable is not enough. > And, more importantly, what name do you use when you need something > more specific than "iterable", but less specific than "sequence"?as in > the glossary entry for dict views, for example? As I said, I would use the term "dict views". If you mean how "dict views" could be defined then I've already linked more than once to the corresponding Python docs issue with the patch http://bugs.python.org/issue25286 >> I don't remember ever using "non-iterator iterable". > > Why would you expect to remember using it, when you're replying to a > message where I invented it for lack of an established better name > (and in hopes that someone would come up with one)? > >> "non-iterator >> iterable" does not qualify as more specific. You need to introduce new >> requirements to the type for that. > > There are things that are iterables, that are not non-iterator > iterables, but the reverse is not true. It's a name for a strict > subset. Which means it's more specific. Could you provide a non-hypothetical practical example from existing code of a function that accepts arbitrary iterables but rejects iterators? Perhaps discussing a specific example would help to iron out the terminology. From 4kir4.1i at gmail.com Fri Oct 2 04:13:39 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Fri, 02 Oct 2015 05:13:39 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> Message-ID: <87io6q10h8.fsf@gmail.com> Steven D'Aprano writes: ... >> You've got the idea: the word *iterable* may be used in the context when >> not all iterables are accepted. > > Sure. But the distinction is that while there are a whole lot of > different iterables: > > - iterables with a sufficiently small number of items > - iterables of hashable items > - iterables of (hashable key, item) pairs > - iterables of prime numbers less than one million > - iterables of strings containing exactly 1 vowel > > etc they are special cases and don't need specialised names. But there > is a *general* distinction between two cases: > > - iterables which are iterators > - iterables which are not iterators > > We have a name for the first set: "iterators". But we don't have a name > for the second set. Andrew suggested "non-iterator iterables" is too > clumsy for general use, and suggests we need a better name. You > suggested "iterables", but that clearly cannot work, since iterators are > a kind of iterable. I meant that you won't use the word *iterable* if your function accepts _only_ iterators and therefore if I see *iterable" I expect that the function can handle arbitrary iterables, not just iterators. If your function rejects iterators then in practice it means that you might want a re-iterable/re-runnable (that I see mentioned in the thread) iterable (not an arbitrary "non-iterator iterable"). If there is no restriction that the same items should be produced the second time ("rerunnable iterable"?) then *collection" (introduced at the top of the thread) may work. Though *collection* implies that an iterable can't return the same (non-self) iterator -- otherwise an iterator is also a collection. I still don't see a practical need to avoid the word "iterable" unless new requirements (in addition to being non-iterator) are present. From steve at pearwood.info Fri Oct 2 04:57:12 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Oct 2015 12:57:12 +1000 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> Message-ID: <20151002025712.GH23642@ando.pearwood.info> On Thu, Oct 01, 2015 at 03:57:18PM -0500, Ron Adam wrote: > On 10/01/2015 02:59 AM, Nick Coghlan wrote: > > >As a result, one interesting way of looking at this problem is to > >ask: what if we *only* offered the conditional operations, *without* > >offering a binary null-coalescing operator? > > How languages treat None, null, and nil, are very fundamental aspects of > how they work. For example I noticed that if we add an None specific > 'or' operator, we may want a None specific 'and' operator too. Why would we do that? What would it do and which other languages have it? > And in order to use those in conditional statements, it makes sense to > have None specific 'if' and 'while'. Which other languages have these? > So it may be a slippery slope. (Or > they may be good to have. I'm very undecided still.) Most slippery-slope arguments aren't actually that slippery. Sure, we could invent a new short-cut for "if spam is None" statement, but it doesn't buy us much benefit, so why would we bother? if spam.eggs[cheese] is None: ... ifn spam.eggs[cheese]: ... You save a few characters typing, big deal. But with an expression, you can save complexity, by avoiding temporary variables: # before expensive = spam.eggs[cheese] # don't calculate this twice value = expensive.attribute if expensive is not None else None # after value = spam.eggs[cheese]?.attribute [...] > Another possibility is to have a *special magic callable* that when > called skips the argument evaluation and returns None. > > NoneCall(these, args, are, never, evaluated) That cannot work with standard Python semantics. In Python, the function doesn't evaluate the arguments, they are evaluated before the function even gets called. For this to work, the parser would have to recognise this NoneCall function *at compile time*, so it can decide whether or not to evaluate the arguments. That is, in general, an impossible task: func = random.choice([NoneCall, lambda *args: None]) func(a+1, b*2, c**3) # to evaluate or not to evaluate? Python has only one thing even remotely like this, the special handling of "super" inside functions, and that's a special case which I think Guido has ruled won't be repeated. But even there, it isn't a change in what gets evaluated when. > Then a special method on objects, obj __cond__call__() can return a > NoneCall to skip the arguments on the right side of ?(, or return > another callable to use them. No, it can't work anything like that. It can't be a regular method call, because the arguments are already evaluated before the method call is made. It has to be a change in what code gets compiled. -- Steve From steve at pearwood.info Fri Oct 2 05:49:36 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Oct 2015 13:49:36 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <87io6q10h8.fsf@gmail.com> References: <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <87io6q10h8.fsf@gmail.com> Message-ID: <20151002034936.GI23642@ando.pearwood.info> On Fri, Oct 02, 2015 at 05:13:39AM +0300, Akira Li wrote: > I still don't see a practical need to avoid the word "iterable" unless > new requirements (in addition to being non-iterator) are present. I don't think anyone has suggested that we should avoid the word iterable. At most, some have suggested that we don't have a good word for those iterables which are not iterators. -- Steve From steve at pearwood.info Fri Oct 2 05:58:33 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 2 Oct 2015 13:58:33 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: References: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <20151001193709.GF23642@ando.pearwood.info> Message-ID: <20151002035833.GJ23642@ando.pearwood.info> On Thu, Oct 01, 2015 at 04:13:37PM -0700, Chris Barker wrote: > On Thu, Oct 1, 2015 at 12:37 PM, Steven D'Aprano > wrote: > > > This makes a difference. Just a few days ago, somebody mis-reported a > > supposed "bug" in all() and any(). For example: > > > > values = (x%5 == 3 for x in range(8)) > > print(list(values)) > > print(all(values)) # should return False > > > > Obvious error is obvious: having printed out the values from the > > generator expression, values is now exhausted, and all() of the empty > > set is True (vacuous truth). The difference between general iterables > > which may or may not be one-shot iterators, and those which are > > definitely not iterators, is not always just an implementation detail. > > > they used a generator expression, when they clearly wanted a list > comprehension -- so yes, it matters what they were getting, I don't know > that adding more vocabulary would help prevent people from making that > mistake... > > if they had been smart enough to call the list() again, before claiming > there was a bug in all -- it may have explained itself. You're missing the point. Don't focus on the fact that the bug was in their understanding of what their code did. Let's just pretend that their *intentional* algorithm was: def alg(iterable): print(list(iterable)) print(all(iterable)) and for the sake of bring this never-ending thread to an end, let's agree for the sake of argument that it cannot be re-written in any other way. Since the semantics of the function are intentional and correct, the parameter is named misleadingly. *iterable* is not sufficiently precise, because the function does not accept any old iterable -- it fails to work correctly on *iterators*, are a sub-kind of iterable. If you want a more practical example, any algorithm which needs to iterate over an interable two or more times needs to specify "iterable which is not an iterator". -- Steve From rosuav at gmail.com Fri Oct 2 06:02:09 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Oct 2015 14:02:09 +1000 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151002035833.GJ23642@ando.pearwood.info> References: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <20151001193709.GF23642@ando.pearwood.info> <20151002035833.GJ23642@ando.pearwood.info> Message-ID: On Fri, Oct 2, 2015 at 1:58 PM, Steven D'Aprano wrote: > If you want a more practical example, any algorithm which needs to > iterate over an interable two or more times needs to specify "iterable > which is not an iterator". For that particular case, I'd reiterate what others have suggested, and use the term "reiterable" for something you can iterate over more than once and get the same results. Sequences are normally reiterable. Any object whose __iter__ is a generator function with stable results will be reiterable. An iterator is not; nor is an open file object, or any other object where iteration consumes an external resource. This seems reasonable. ChrisA From brenbarn at brenbarn.net Fri Oct 2 06:05:59 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Thu, 01 Oct 2015 21:05:59 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151002035833.GJ23642@ando.pearwood.info> References: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <20151001193709.GF23642@ando.pearwood.info> <20151002035833.GJ23642@ando.pearwood.info> Message-ID: <560E02A7.6050604@brenbarn.net> On 2015-10-01 20:58, Steven D'Aprano wrote: > Since the semantics of the function are intentional and correct, the > parameter is named misleadingly. *iterable* is not sufficiently precise, > because the function does not accept any old iterable -- it fails to > work correctly on *iterators*, are a sub-kind of iterable. > > If you want a more practical example, any algorithm which needs to > iterate over an interable two or more times needs to specify "iterable > which is not an iterator". I would disagree with this, because this terminology is both too technical and not technical enough. Just because something isn't an iterator doesn't mean you can iterate it multiple times. You *could* write an iterable which is not an iterator but still can't be iterated over multiple times (because, say, it returns a reference to some stored iterator that can't be restarted, or because it creates a custom iterator that references some persistent state of the iterable). If what you want is an iterable that can be iterated multiple times, then just say that. (Or say "reiterable" or "reentrant iterable" or whatever.) There's no need to bring iterators into it at all. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From brenbarn at brenbarn.net Fri Oct 2 06:06:51 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Thu, 01 Oct 2015 21:06:51 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151002035833.GJ23642@ando.pearwood.info> References: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <20151001193709.GF23642@ando.pearwood.info> <20151002035833.GJ23642@ando.pearwood.info> Message-ID: <560E02DB.50305@brenbarn.net> On 2015-10-01 20:58, Steven D'Aprano wrote: > > Since the semantics of the function are intentional and correct, the > > parameter is named misleadingly. *iterable* is not sufficiently precise, > > because the function does not accept any old iterable -- it fails to > > work correctly on *iterators*, are a sub-kind of iterable. > > > > If you want a more practical example, any algorithm which needs to > > iterate over an interable two or more times needs to specify "iterable > > which is not an iterator". I would disagree with this, because this terminology is both too technical and not technical enough. Just because something isn't an iterator doesn't mean you can iterate it multiple times. You *could* write an iterable which is not an iterator but still can't be iterated over multiple times (because, say, it returns a reference to some stored iterator that can't be restarted, or because it creates a custom iterator that references some persistent state of the iterable). If what you want is an iterable that can be iterated multiple times, then just say that. (Or say "reiterable" or "reentrant iterable" or whatever.) There's no need to bring iterators into it at all. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From 4kir4.1i at gmail.com Fri Oct 2 08:30:23 2015 From: 4kir4.1i at gmail.com (Akira Li) Date: Fri, 02 Oct 2015 09:30:23 +0300 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence References: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <20151001193709.GF23642@ando.pearwood.info> <20151002035833.GJ23642@ando.pearwood.info> Message-ID: <87bnch235s.fsf@gmail.com> Steven D'Aprano writes: > On Thu, Oct 01, 2015 at 04:13:37PM -0700, Chris Barker wrote: >> On Thu, Oct 1, 2015 at 12:37 PM, Steven D'Aprano >> wrote: >> >> > This makes a difference. Just a few days ago, somebody mis-reported a >> > supposed "bug" in all() and any(). For example: >> > >> > values = (x%5 == 3 for x in range(8)) >> > print(list(values)) >> > print(all(values)) # should return False >> > >> > Obvious error is obvious: having printed out the values from the >> > generator expression, values is now exhausted, and all() of the empty >> > set is True (vacuous truth). The difference between general iterables >> > which may or may not be one-shot iterators, and those which are >> > definitely not iterators, is not always just an implementation detail. >> >> >> they used a generator expression, when they clearly wanted a list >> comprehension -- so yes, it matters what they were getting, I don't know >> that adding more vocabulary would help prevent people from making that >> mistake... >> >> if they had been smart enough to call the list() again, before claiming >> there was a bug in all -- it may have explained itself. > > You're missing the point. Don't focus on the fact that the bug was in > their understanding of what their code did. Let's just pretend that > their *intentional* algorithm was: > > def alg(iterable): > print(list(iterable)) > print(all(iterable)) > > and for the sake of bring this never-ending thread to an end, let's > agree for the sake of argument that it cannot be re-written in any other > way. > > Since the semantics of the function are intentional and correct, the > parameter is named misleadingly. *iterable* is not sufficiently precise, > because the function does not accept any old iterable -- it fails to > work correctly on *iterators*, are a sub-kind of iterable. list(iterable) does not work for infinite iterables. set(iterable) does not work for iterables that yield non-hashable items. dict(iterable) does not work for iterables that do not yield key,value pairs. Each builtin specifies what type of iterable it accepts but the parameter IS called *iterable*. Are you suggesting to rename the parameters? > If you want a more practical example, any algorithm which needs to > iterate over an interable two or more times needs to specify "iterable > which is not an iterator". If the intent is to write: def alg(iterable): seq = list(iterable) print(seq) print(all(seq)) You could say that the 1st alg() accepts a "repeatable deterministic iterable". From tjreedy at udel.edu Fri Oct 2 09:19:24 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 2 Oct 2015 03:19:24 -0400 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <87612r2koz.fsf@gmail.com> References: <560C25B4.5050000@egenix.com> <147B2DEC-310B-498A-B468-03F0053F55B7@yahoo.com> <560C2D52.3080809@egenix.com> <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87a8s32llk.fsf@gmail.com> <87612r2koz.fsf@gmail.com> Message-ID: On 10/1/2015 1:59 AM, Akira Li wrote: > Random832 writes: > >> Akira Li <4kir4.1i at gmail.com> writes: >>> I don't remember ever using "non-iterator iterable". "non-iterator >>> iterable" does not qualify as more specific. You need to introduce new >>> requirements to the type for that. >> >> The question is, how do you *simply* state the very common requirement >> for an iterable to not behave in a specific undesirable way that all >> iterators do, and that it is very uncommon for any iterable other than >> an iterator to do? Are you opposed to having a word for this concept at >> all, or do you just not like the terms other people are suggesting? > > That term is **iterable**. As I already said: > > Specific application may use more specific requirements e.g.: > > list(iterable): finite_iterable > - does it mean that all iterables must be finite? > - do we need a special word to describe what list() accepts? > > set(iterable): iterable_of_hashables > - does it mean that all iterables must yield hashable items? > - do we need a special word to describe what set() accepts? > > dict(iterable): iterable_of_pairs (whose first member is hashable) > - does it mean that all iterables must yield pairs? > - do we need a special word to describe what dict() accepts? Whatever word is used in the signature, the description should be in the doc and docstring, preferable in the first line. Return a list with members from a finite iterable. Return a set with members from an iterable of hashable objects.* Return a string joining items from an iterable of strings. *Also, equality between object should be transitive > You've got the idea: the word *iterable* may be used in the context when > not all iterables are accepted. Right. Each one should be documented properly. -- Terry Jan Reedy From ron3200 at gmail.com Fri Oct 2 12:01:12 2015 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 2 Oct 2015 05:01:12 -0500 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> Message-ID: On 10/01/2015 04:12 PM, Random832 wrote: > On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: >> >And to go out on a limb...;-) >> > >> >Another possibility is to have a*special magic callable* that when >> >called skips the argument evaluation and returns None. > That's dangerous talk indeed. Special magic callables are Lisp > territory.;) It's also lambda calculus territory. > And I don't even know how you'd implement it efficiently without them > being known at compile-time. > I guess at*every* callsite you could test if the callable is magic, and > if it is evaluate the arguments, and if it's not just pass in a lambda > that will return the arguments. But you've got to generate those lambdas > for*all* callsites, even the vast majority that will never be a magic > callable. I realized this evening the parser doesn't need to know at parse time, and the object doesn't need to be special. It's the syntax that gives it the specialness, not the object. So what I was thinking is still possible, but it would work more like the other suggestions. If you look at byte code generated for a function call... >>> def bar(x, y): ... return x + y ... >>> def foo(x, y): ... return bar(x+1, y+1) ... >>> dis(foo) 2 0 LOAD_GLOBAL 0 (bar) 3 LOAD_FAST 0 (x) 6 LOAD_CONST 1 (1) 9 BINARY_ADD 10 LOAD_FAST 1 (y) 13 LOAD_CONST 1 (1) 16 BINARY_ADD 17 CALL_FUNCTION 2 (2 positional, 0 keyword pair) 20 RETURN_VALUE You will notice the function is loaded on the stack *before* the argument expressions are evaluated. It won't require generating lamba expressions, just a conditional jump where '?(' is used. So no, it won't be required at every call site. Something roughly like ... (presume the index's are adjusted correctly) 2 0 LOAD_GLOBAL 0 (bar) #-- inserted only if '?(' is used -------------- POP_JUMP_IF_NOT_NOCALL 3 LOAD_CONST 0 (None) JUMP_FORWARD 8 (to 20) #----------------------------------------------- 3 LOAD_FAST 0 (x) 6 LOAD_CONST 1 (1) 9 BINARY_ADD 10 LOAD_FAST 1 (y) 13 LOAD_CONST 1 (1) 16 BINARY_ADD 17 CALL_FUNCTION 2 (2 positional, 0 keyword pair) 20 RETURN_VALUE So the parser doesn't need to know if it's the NoneCall object at parse time, it just needs to know to check for it, and the ?( syntax can tell it to do that only where it's needed. There really isn't anything special about the NoneCall object, it's just a reserved builtin object. And adding the check for it isn't all the special either. It's the behavior that is special because of the ?( sytnax. I was thinking it could call a __cond_call__ method on the object. That extra indirect call could allow objects to specify how they respond to ?( conditional calls. It's just requires an extra LOAD_ATTR and call it with no arguments in the bytecode. The case of unpacking the args if the object is None would require an additional check for None and to load an identity function in it's place or could be done by adding a __cond_call__ method (if that is how ?( will work) to None that is an identity function. value = obj?(default) #None?(default) --> identity(default) Think of these as moving conditions that would otherwise be elsewhere to a more convenient location. So I don't expect them to be much slower than the equivalent code that does the same thing. A __cond_call__ method works because it can't conflict with __call__. So it avoids any conflicts that might occur by calling __call__ by mistake. The point is that a conditional call syntax could also fill the need of the or None operator. The specific details may be a bit different than what I describe here, but I think it's doable. > What if the callable only wants*some* of the arguments? Hey, if this > had existed back then the ternary operator could have been a normal > function - instead of (b() if a else c()) just do iif(a, b(), c()). Only wanting some arguments would be much harder. It might be done by splitting the arguments into nested ?( calls similar to how we apply multiple decorators. I don't expect it to be pretty or practical, so lets just not go there for now. > And is this going to be fully general? I.e. should this be supported for > regular operators? If __add__ is magic does + do this, for example? No, it would only work with ?( syntax. Cheers, Ron From abarnert at yahoo.com Fri Oct 2 12:21:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 2 Oct 2015 03:21:43 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> Message-ID: <2A88A715-4547-4166-9F52-5B467FF9B174@yahoo.com> On Oct 2, 2015, at 03:01, Ron Adam wrote: > >> On 10/01/2015 04:12 PM, Random832 wrote: >>> On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: >>> >And to go out on a limb...;-) >>> > >>> >Another possibility is to have a*special magic callable* that when >>> >called skips the argument evaluation and returns None. > >> That's dangerous talk indeed. Special magic callables are Lisp >> territory.;) > > It's also lambda calculus territory. > >> And I don't even know how you'd implement it efficiently without them >> being known at compile-time. > >> I guess at*every* callsite you could test if the callable is magic, and >> if it is evaluate the arguments, and if it's not just pass in a lambda >> that will return the arguments. But you've got to generate those lambdas >> for*all* callsites, even the vast majority that will never be a magic >> callable. > > I realized this evening the parser doesn't need to know at parse time, and the object doesn't need to be special. It's the syntax that gives it the specialness, not the object. So what I was thinking is still possible, but it would work more like the other suggestions. But if you've already got special handling for ?(), you don't need this NoCall object at all. If bar is None, ?() just skips the call. Compare the bytecode for your version: > > 2 0 LOAD_GLOBAL 0 (bar) > > #-- inserted only if '?(' is used -------------- > POP_JUMP_IF_NOT_NOCALL 3 > LOAD_CONST 0 (None) > JUMP_FORWARD 8 (to 20) > #----------------------------------------------- ... to the simple ?() check version: 2 0 LOAD_GLOBAL 0 (bar) #-- inserted only if '?(' is used JUMP_IF_NONE 8 (to 20) #-------------------------------- That's it. (And if you don't want to add a JUMP_IF_NONE opcode, you just DUP, compare to None, and POP_JUMP_IF_TRUE.) Just like ?. and ?[], the LOAD_* is followed by a JUMP_IF_NONE to the end of the first call (or the end of the entire primary if there is no call, but that case can't come up for ?() because it is a call), and nothing else has to be done. > The case of unpacking the args if the object is None would require an additional check for None and to load an identity function in it's place or could be done by adding a __cond_call__ method (if that is how ?( will work) to None that is an identity function. Why do you want to do this? And what does it even mean to "unpack the args" here? Does the identity function return a tuple and a dict for the positional and keyword functions respectively? When would you ever want that? Also, to call the identity function (or anything else) on the args requires evaluating them, which defeats the entire purpose of short-circuiting evaluation of the args. And, even if the idea of a universal identity function made sense here, why even call it instead of just not calling anything? What's the benefit from that extra work? -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Fri Oct 2 14:54:46 2015 From: random832 at fastmail.com (Random832) Date: Fri, 02 Oct 2015 08:54:46 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56097AFB.1040906@oddbird.net> <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> Message-ID: <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> On Fri, Oct 2, 2015, at 06:01, Ron Adam wrote: > On 10/01/2015 04:12 PM, Random832 wrote: > > On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: > >> >And to go out on a limb...;-) > >> > > >> >Another possibility is to have a*special magic callable* that when > >> >called skips the argument evaluation and returns None. > > > That's dangerous talk indeed. Special magic callables are Lisp > > territory.;) > > It's also lambda calculus territory. Does lambda calculus really have a notion of two types of function that are called with the same syntax but one receives the values as they have been evaluated and the other receives some abstract representation of how to get them if it needs them? That doesn't even make any sense. Does it even have a notion of side effects and therefore when/whether things are evaluated if they are not needed? Not everything in Lisp is in lambda calculus. > I realized this evening the parser doesn't need to know at parse time, > and the object doesn't need to be special. It's the syntax that gives > it the specialness, not the object. So what I was thinking is still > possible, but it would work more like the other suggestions. > > If you look at byte code generated for a function call... > > You will notice the function is loaded on the stack *before* the > argument expressions are evaluated. > > It won't require generating lamba expressions, just a conditional jump > where '?(' is used. So no, it won't be required at every call site. What does this have to do with the idea of a magic callable? If this is implemented with a magic callable then A) *any* callable might be a magic callable and B) the idea original idea strongly implied a generalized notion of magic callables, of which NoneCall would be only one example. How do you pass the arguments to the *other* magic callables that *do* [maybe] evaluate them? You've basically just explained how the bytecode works for ?( if it is *not* implemented with a magic callable. So how does making it NoneCall instead of just None improve anything? From srkunze at mail.de Fri Oct 2 16:07:13 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 02 Oct 2015 16:07:13 +0200 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <20151002034936.GI23642@ando.pearwood.info> References: <08E63279-5DFF-4F7A-9B7B-B927D34BC4FA@yahoo.com> <560C3C45.1070107@egenix.com> <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <87io6q10h8.fsf@gmail.com> <20151002034936.GI23642@ando.pearwood.info> Message-ID: <560E8F91.4000209@mail.de> On 02.10.2015 05:49, Steven D'Aprano wrote: > At most, some have suggested that we don't have a good > word for those iterables which are not iterators. I sometimes use re-usable iterables. From rymg19 at gmail.com Fri Oct 2 16:13:19 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 02 Oct 2015 09:13:19 -0500 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> References: <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> Message-ID: On October 2, 2015 7:54:46 AM CDT, Random832 wrote: >On Fri, Oct 2, 2015, at 06:01, Ron Adam wrote: >> On 10/01/2015 04:12 PM, Random832 wrote: >> > On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: >> >> >And to go out on a limb...;-) >> >> > >> >> >Another possibility is to have a*special magic callable* that >when >> >> >called skips the argument evaluation and returns None. >> >> > That's dangerous talk indeed. Special magic callables are Lisp >> > territory.;) >> >> It's also lambda calculus territory. > >Does lambda calculus really have a notion of two types of function that >are called with the same syntax but one receives the values as they >have >been evaluated and the other receives some abstract representation of >how to get them if it needs them? That doesn't even make any sense. >Does >it even have a notion of side effects and therefore when/whether things >are evaluated if they are not needed? > Indeed. Lambda calculus is untyped... >Not everything in Lisp is in lambda calculus. > >> I realized this evening the parser doesn't need to know at parse >time, >> and the object doesn't need to be special. It's the syntax that >gives >> it the specialness, not the object. So what I was thinking is still >> possible, but it would work more like the other suggestions. >> >> If you look at byte code generated for a function call... >> >> You will notice the function is loaded on the stack *before* the >> argument expressions are evaluated. >> >> It won't require generating lamba expressions, just a conditional >jump >> where '?(' is used. So no, it won't be required at every call site. > >What does this have to do with the idea of a magic callable? If this is >implemented with a magic callable then A) *any* callable might be a >magic callable and B) the idea original idea strongly implied a >generalized notion of magic callables, of which NoneCall would be only >one example. How do you pass the arguments to the *other* magic >callables that *do* [maybe] evaluate them? > >You've basically just explained how the bytecode works for ?( if it is >*not* implemented with a magic callable. So how does making it NoneCall >instead of just None improve anything? >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From srkunze at mail.de Fri Oct 2 16:20:02 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 02 Oct 2015 16:20:02 +0200 Subject: [Python-ideas] Binary f-strings In-Reply-To: <56089692.1080303@trueblade.com> References: <56089692.1080303@trueblade.com> Message-ID: <560E9292.6040304@mail.de> On 28.09.2015 03:23, Eric V. Smith wrote: > The only real question is: what encoding to use for the second parameter > to bytes()? Since an object must return unicode from __format__(), I > need to convert that to bytes in order to join everything together. But how? Cf. https://www.python.org/dev/peps/pep-0461/#interpolation It says: b"%x" % val is equivalent to: ("%x" % val).encode("ascii") So, ASCII would make a lot of sense to me as well. > Here I suggest 'ascii'. Unfortunately, this would give an error if > __format__ returned anything with a char greater than 127. I think we've > learned that an API that only raises an exception with certain specific > inputs is fragile. Could you be more specific here? Best, Sven From eric at trueblade.com Fri Oct 2 16:26:49 2015 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 02 Oct 2015 10:26:49 -0400 Subject: [Python-ideas] Binary f-strings In-Reply-To: <560E9292.6040304@mail.de> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> Message-ID: <560E9429.9040205@trueblade.com> On 10/02/2015 10:20 AM, Sven R. Kunze wrote: > On 28.09.2015 03:23, Eric V. Smith wrote: >> Here I suggest 'ascii'. Unfortunately, this would give an error if >> __format__ returned anything with a char greater than 127. I think we've >> learned that an API that only raises an exception with certain specific >> inputs is fragile. > > Could you be more specific here? bf'{foo}' Might succeed or fail, depending on what foo returns for __format__. If foo is 'bar', it succeeds. If it's '\u1234', it fails. But some of the other arguments are making me think bf'' is a bad idea, so now I'm leaning towards not implementing it. Eric. From jbvsmo at gmail.com Fri Oct 2 16:48:07 2015 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Fri, 2 Oct 2015 11:48:07 -0300 Subject: [Python-ideas] Binary f-strings In-Reply-To: <560E9429.9040205@trueblade.com> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> Message-ID: > > > But some of the other arguments are making me think bf'' is a bad idea, > so now I'm leaning towards not implementing it. > > What about rf''? (sorry for being off topic here) Regex could benefit from it: my_regex = rf"?\w+\s*({'|'.join(expected_words)})$" -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Oct 2 16:52:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Oct 2015 00:52:23 +1000 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> Message-ID: On Sat, Oct 3, 2015 at 12:48 AM, Jo?o Bernardo wrote: >> But some of the other arguments are making me think bf'' is a bad idea, >> so now I'm leaning towards not implementing it. >> > > What about rf''? (sorry for being off topic here) > > Regex could benefit from it: > > my_regex = rf"?\w+\s*({'|'.join(expected_words)})$" > Works fine: rosuav at sikorsky:~$ python3 Python 3.6.0a0 (default:48943533965e, Sep 28 2015, 11:27:38) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> expected_words = ["foo", "bar"] >>> my_regex = rf"?\w+\s*({'|'.join(expected_words)})$" >>> print(my_regex) ?\w+\s*(foo|bar)$ >>> ChrisA From eric at trueblade.com Fri Oct 2 16:52:14 2015 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 02 Oct 2015 10:52:14 -0400 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> Message-ID: <560E9A1E.5020409@trueblade.com> On 10/02/2015 10:48 AM, Jo?o Bernardo wrote: > > But some of the other arguments are making me think bf'' is a bad idea, > so now I'm leaning towards not implementing it. > > > What about rf''? (sorry for being off topic here) > > Regex could benefit from it: > > my_regex = rf"?\w+\s*({'|'.join(expected_words)})$" > That's already implemented. Its use in regular expressions is mentioned in the PEP. From srkunze at mail.de Fri Oct 2 17:43:46 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 02 Oct 2015 17:43:46 +0200 Subject: [Python-ideas] Binary f-strings In-Reply-To: <560E9429.9040205@trueblade.com> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> Message-ID: <560EA632.80805@mail.de> On 02.10.2015 16:26, Eric V. Smith wrote: > bf'{foo}' > > Might succeed or fail, depending on what foo returns for __format__. If > foo is 'bar', it succeeds. If it's '\u1234', it fails. I know a lot of functions that fail when passing the wrong kind of arguments. What's so wrong with it? > But some of the other arguments are making me think bf'' is a bad idea, > so now I'm leaning towards not implementing it. I see. Do you think of an alternative solution? I was digging deeper into the matter of binary/byte strings formatting in order to sympathise why {} is not usable for binary protocols. Let's look at this practical example https://blog.tox.chat/2015/09/fuzzing-the-new-groupchats/ I hope the tox protocol fully qualifies as a wireframe protocol. What he's trying to do it is to fuzzing his new groupchat implementation by creating more-or-less random packages and feed them into tox core to see if it breaks. He conveniently uses this type of syntax to describe the structure of the first header: Header 1: [ Packet ID (1 b) | Chat ID hash (4 b) | Sender PK (32 b) | nonce (24 b) ] Interested in writing a fuzzer, I would find the following really helpful as it mirrors the description within his blog post: header1 = bf'{packet_id}{chat_id}{sender_pk}{nonce}' # which should be the same as header1 = b'%b%b%b%b' % (packet_id, chat_id, sender_pk, nonce) I wouldn't mind specifying the encoding for all non-byte-string arguments. Why? Because I would be working with bytes anyway, so no formatting (as in format()) would be necessary in the first place. However, I like the syntax for specifying the structure of (byte) strings. Does this makes sense? Best, Sven From guido at python.org Fri Oct 2 18:00:56 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Oct 2015 09:00:56 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: <560EA632.80805@mail.de> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: Bingo. IMO the exact same arguments that show why f'{x} {y}' is better than '%s %s' % (x, y) applies to byte strings. It would be totally acceptable if it only took bytes (and bytearray, and memoryview) and numbers (which we can guarantee are rendered in ASCII only). On Fri, Oct 2, 2015 at 8:43 AM, Sven R. Kunze wrote: > On 02.10.2015 16:26, Eric V. Smith wrote: > >> bf'{foo}' >> >> Might succeed or fail, depending on what foo returns for __format__. If >> foo is 'bar', it succeeds. If it's '\u1234', it fails. >> > > I know a lot of functions that fail when passing the wrong kind of > arguments. What's so wrong with it? > > But some of the other arguments are making me think bf'' is a bad idea, >> so now I'm leaning towards not implementing it. >> > > I see. Do you think of an alternative solution? > > > I was digging deeper into the matter of binary/byte strings formatting in > order to sympathise why {} is not usable for binary protocols. Let's look > at this practical example > https://blog.tox.chat/2015/09/fuzzing-the-new-groupchats/ I hope the tox > protocol fully qualifies as a wireframe protocol. What he's trying to do it > is to fuzzing his new groupchat implementation by creating more-or-less > random packages and feed them into tox core to see if it breaks. > > He conveniently uses this type of syntax to describe the structure of the > first header: > > Header 1: [ Packet ID (1 b) | Chat ID hash (4 b) | Sender PK (32 > b) | nonce (24 b) ] > > > Interested in writing a fuzzer, I would find the following really helpful > as it mirrors the description within his blog post: > > header1 = bf'{packet_id}{chat_id}{sender_pk}{nonce}' > > # which should be the same as > > header1 = b'%b%b%b%b' % (packet_id, chat_id, sender_pk, nonce) > > I wouldn't mind specifying the encoding for all non-byte-string arguments. > Why? Because I would be working with bytes anyway, so no formatting (as in > format()) would be necessary in the first place. However, I like the syntax > for specifying the structure of (byte) strings. > > > Does this makes sense? > > > Best, > Sven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Fri Oct 2 19:47:53 2015 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 2 Oct 2015 12:47:53 -0500 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> References: <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> Message-ID: On 10/02/2015 07:54 AM, Random832 wrote: > On Fri, Oct 2, 2015, at 06:01, Ron Adam wrote: >> On 10/01/2015 04:12 PM, Random832 wrote: >>> On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: >>>>> And to go out on a limb...;-) >>>>> >>>>> Another possibility is to have a*special magic callable* that when >>>>> called skips the argument evaluation and returns None. >> >>> That's dangerous talk indeed. Special magic callables are Lisp >>> territory.;) >> >> It's also lambda calculus territory. > > Does lambda calculus really have a notion of two types of function that > are called with the same syntax but one receives the values as they have > been evaluated and the other receives some abstract representation of > how to get them if it needs them? That doesn't even make any sense. Does > it even have a notion of side effects and therefore when/whether things > are evaluated if they are not needed? > > Not everything in Lisp is in lambda calculus. I wasn't really comparing it to lisp, but I understood your reference to it. The comparison to lambda calculus is very loose also. BTW, in lisp (and scheme) there are a handful of special functions. One of those is the if function which avoids early evaluation of the expressions till after the condition is evaluated. It can be implemented as a normal lisp expression but not without quoting the arguments. >> I realized this evening the parser doesn't need to know at parse time, >> and the object doesn't need to be special. It's the syntax that gives >> it the specialness, not the object. So what I was thinking is still >> possible, but it would work more like the other suggestions. >> >> If you look at byte code generated for a function call... >> >> You will notice the function is loaded on the stack *before* the >> argument expressions are evaluated. >> >> It won't require generating lamba expressions, just a conditional jump >> where '?(' is used. So no, it won't be required at every call site. > > What does this have to do with the idea of a magic callable? As I said above it doesn't need to magic after all. > If this is > implemented with a magic callable then A) *any* callable might be a > magic callable and B) the idea original idea strongly implied a > generalized notion of magic callables, of which NoneCall would be only > one example. How do you pass the arguments to the *other* magic > callables that *do* [maybe] evaluate them? > > You've basically just explained how the bytecode works for ?( if it is > *not* implemented with a magic callable. So how does making it NoneCall > instead of just None improve anything? It's not quite the same. The proposed ?( skips the arguments if the right hand side is None. None?(...) ---> None So you can't use that to supply a default directly. value = None?(default) # returns None (not useful here) To make that work, think of applying None to the arguments. Which in turn does nothing to them, they just get returned. value = None?(default) --> default But to get the inverse of that which is the original preposed ?( behavior, you need a different way to trigger that. We could use False I suppose. value = False?(default) --> None And of course.... value = other?(default) --> other(default) This is just based on the thought that a conditional ?( calling syntax could be useful in a broader scope and not just used as a None coalescing operator. So in the above, None, False, and other can be replaced by expressions. def when(x): return None if x else False value = when(cond)?(expr) # expr or None With the original proposed ?( operator the set default example is still doable, but it needs a helper function. None?(expr) --> None other?(expr) --> other(expr) # The True case def is_none(cond): return None if cond is None else lambda x:x value = is_none(value)?(default_value) Compared to value = default_value if expr is None else None I was trying to shorten it to.. value = expr?(default) But maybe that just won't work in a nice enough way. Well it's just a thought/suggestion. (shrug) Cheers, Ron From abarnert at yahoo.com Fri Oct 2 21:56:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 2 Oct 2015 12:56:47 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <5609985C.40603@oddbird.net> <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> Message-ID: On Oct 2, 2015, at 10:47, Ron Adam wrote: > >> On 10/02/2015 07:54 AM, Random832 wrote: >>> On Fri, Oct 2, 2015, at 06:01, Ron Adam wrote: >>>> On 10/01/2015 04:12 PM, Random832 wrote: >>>> On Thu, Oct 1, 2015, at 16:57, Ron Adam wrote: >>>>>> And to go out on a limb...;-) >>>>>> >>>>>> Another possibility is to have a*special magic callable* that when >>>>>> called skips the argument evaluation and returns None. >>> >>>> That's dangerous talk indeed. Special magic callables are Lisp >>>> territory.;) >>> >>> It's also lambda calculus territory. >> >> Does lambda calculus really have a notion of two types of function that >> are called with the same syntax but one receives the values as they have >> been evaluated and the other receives some abstract representation of >> how to get them if it needs them? That doesn't even make any sense. Does >> it even have a notion of side effects and therefore when/whether things >> are evaluated if they are not needed? >> >> Not everything in Lisp is in lambda calculus. > > I wasn't really comparing it to lisp, but I understood your reference to it. The comparison to lambda calculus is very loose also. The comparison to Lisp is very good. Lisp has the exact concept you're asking for: a way to define a function that receives the expressions used as its arguments rather than their evaluated values. The comparison to lambda calculus doesn't even make sense. It does even have the concept of the string of lambda calculus symbols (or any other parallel) as a thing that can be passed around. > BTW, in lisp (and scheme) there are a handful of special functions. One of those is the if function which avoids early evaluation of the expressions till after the condition is evaluated. It can be implemented as a normal lisp expression but not without quoting the arguments. Lisp and scheme also let you define new special functions that automatically get the quoted form of their arguments. (Modern Lisps don't usually have fexprs, but they have macros and special-forms, which are close enough for this discussion, I think.) And that's exactly what you're asking for here. The fact that you can define an if special form that doesn't require the caller to quote the arguments means the language doesn't need flow control expressions; you can define them yourself. There are both pros and cons to that, but the cons make it highly unsuitable for Python. >>> I realized this evening the parser doesn't need to know at parse time, >>> and the object doesn't need to be special. It's the syntax that gives >>> it the specialness, not the object. So what I was thinking is still >>> possible, but it would work more like the other suggestions. >>> >>> If you look at byte code generated for a function call... >>> >>> You will notice the function is loaded on the stack *before* the >>> argument expressions are evaluated. >>> >>> It won't require generating lamba expressions, just a conditional jump >>> where '?(' is used. So no, it won't be required at every call site. >> >> What does this have to do with the idea of a magic callable? > > As I said above it doesn't need to magic after all. > >> If this is >> implemented with a magic callable then A) *any* callable might be a >> magic callable and B) the idea original idea strongly implied a >> generalized notion of magic callables, of which NoneCall would be only >> one example. How do you pass the arguments to the *other* magic >> callables that *do* [maybe] evaluate them? >> >> You've basically just explained how the bytecode works for ?( if it is >> *not* implemented with a magic callable. So how does making it NoneCall >> instead of just None improve anything? > > It's not quite the same. The proposed ?( skips the arguments if the right hand side is None. > > None?(...) ---> None > > So you can't use that to supply a default directly. > > value = None?(default) # returns None (not useful here) Again, why would you want that to return default? Imagine you write spam = eggs?(default), hoping to get default if eggs is None. What are you expecting if spam is not None then? > To make that work, think of applying None to the arguments. Which in turn does nothing to them, they just get returned. > > value = None?(default) --> default Functions don't take one argument; they take zero or more positional arguments and zero or more keyword arguments. And there are probably more calls with 0 or 2 arguments than 1 in an average program (at least if you don't count the implicit self in method calls). So, what does it mean to return "the argument"? The most reasonable answer I can imagine is this: def ident(*args, **kw): return args, kw But that means that to actually use it, you have to write something like this: eggs = spam?(default) try: args, kw = default eggs = args[0] except (IndexError, ValueError): pass And even that isn't correct if the normal return value of non-None spam(default) happens to be an iterable of two elements whose first element is a non-empty sequence. And, even if it were correct, it seems a lot less readable than: eggs = spam(default) if spam is not None else default By comparison, returning None no matter what the arguments are makes sense in a way that returning the arguments doesn't: because None is a potentially usable value (especially with further chaining), unlike the arguments pair of collections. > But to get the inverse of that which is the original preposed ?( behavior, you need a different way to trigger that. We could use False I suppose. > > value = False?(default) --> None > > And of course.... > > value = other?(default) --> other(default) > > > > This is just based on the thought that a conditional ?( calling syntax could be useful in a broader scope and not just used as a None coalescing operator. > > So in the above, None, False, and other can be replaced by expressions. > > def when(x): > return None if x else False > > value = when(cond)?(expr) # expr or None > > > > With the original proposed ?( operator the set default example is still doable, but it needs a helper function. > > None?(expr) --> None > > other?(expr) --> other(expr) # The True case > > > def is_none(cond): > return None if cond is None else lambda x:x > > value = is_none(value)?(default_value) > > > Compared to > > value = default_value if expr is None else None > > > > I was trying to shorten it to.. > > value = expr?(default) > > But maybe that just won't work in a nice enough way. > > Well it's just a thought/suggestion. (shrug) > > > Cheers, > Ron From abarnert at yahoo.com Fri Oct 2 22:10:30 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 2 Oct 2015 13:10:30 -0700 Subject: [Python-ideas] Consider making enumerate a sequence if its argument is a sequence In-Reply-To: <87bnch235s.fsf@gmail.com> References: <9E2C1054-551B-41B1-A0D1-2E54A1FA8BF3@yahoo.com> <87vbar300y.fsf@gmail.com> <87r3lf2vl4.fsf@gmail.com> <87bncj2mqa.fsf@gmail.com> <20151001151028.GD23642@ando.pearwood.info> <20151001193709.GF23642@ando.pearwood.info> <20151002035833.GJ23642@ando.pearwood.info> <87bnch235s.fsf@gmail.com> Message-ID: <58A4DFE1-0ED5-4ED0-90E2-2F3346F64E40@yahoo.com> On Oct 1, 2015, at 23:30, Akira Li <4kir4.1i at gmail.com> wrote: > > Each builtin specifies what type of iterable it accepts but the > parameter IS called *iterable*. Are you suggesting to rename the > parameters? I think this is where the confusion arises. No one but you is talking about parameter names. The issue is just about having consistent, well-understood names that can be used in documentation, discussions like this thread, answers on the main Python list or StackOverflow, etc. The question of exactly which things we need to name (non-iterator iterable, repeatable iterables, repeatable iterables that always iterate the same elements, iterables that don't return self when iterates, iterables that always return a new object each time they're iterafed, ?) may be an open question (because you're right that they're not all the same thing, even if they usually overlap), but some subset of those things is relevant often enough that some of them need names. For example, you could tell the user who had the any/all "bug" that "The way you've coded that function, it only works for re-iterables/collections/whatever, but you're passing it the result of a generator expression, which is an iterator." That (together with a place the user can look up re-iterable/collection/whatever--whether that place is the glossary or the collective mind of the community or whatever) explains the problem without the need for a long explanation on the fact that not all iterables are reusable in the way he's trying to reuse them, and in particular iterators never are, and so on. The user doesn't then need to change his parameter name from "tests" to "reiterable" or anything like that--he's already got a perfectly good name. Of course it's not impossible that some of these concepts might also be useful as ABCs and/or static types, in which case it's possible he may want to add an annotation. But that's a separate issue, one which, again, nobody else has raised. From christoph at grothesque.org Fri Oct 2 23:41:30 2015 From: christoph at grothesque.org (Christoph Groth) Date: Fri, 02 Oct 2015 23:41:30 +0200 Subject: [Python-ideas] asyncio: factoring-out of the general mechanism References: <877fn61rmy.fsf@grothesque.org> Message-ID: <87twq9uewl.fsf@grothesque.org> Guido van Rossum wrote: > Well, what else did you have in mind? Remember that until we fix the GIL > (which won't happen until Python 4) you won't get benefit from async > programming unless you are overlapping with I/O, and (due to the > limitations of select/poll/etc.) that is pretty much limited to network and > IPC (disk I/O in particular cannot be overlapped in this way). I recently realized that asynchronous programming is also useful for scientific computation: It allows to quite naturally express a complex algorithm that at various stages launches background computation. A good application of this technique is adaptive numerical integration. See [1] for an example script that uses asyncio. There, the asynchronous master Python process performs the integration algorithm (that is computationally cheap but has quite complex control flow), while the integrand function is evaluated by worker processes. Now, this particular example uses concurrent.futures, but one could also use MPI, scoop, or 0mq. This is what made me wonder about the monolithic design of asyncio. We do not want to add support for all these libraries to the asyncio package? Already now importing asyncio triggers a considerable avalanche of imports that do not all seem necessary for most problems. > Note that the async/await keywords themselves have no direct link to an > event loop -- they are just a different way to spell generators. It's just > that they are only useful with asyncio or some other event loop library. That's clear. I was just wondering whether it would make sense to split up asyncio into a basic package that only provides the event loop itself (and does not depend on any particular protocol), and support for various "applications". The parts that relate to various "applications" could then be added to other modules. For example, the subprocess package could be extended by some coroutines for launching processes. But perhaps coroutines (or at least their Python flavor) only make sense within the context of a specific event loop and the language does not want to mandate the use of a specific one? Then indeed it seems to be a good idea to structure async functionality around specific event loops. Christoph -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 810 bytes Desc: not available URL: From christoph at grothesque.org Fri Oct 2 23:47:22 2015 From: christoph at grothesque.org (Christoph Groth) Date: Fri, 02 Oct 2015 23:47:22 +0200 Subject: [Python-ideas] asyncio: factoring-out of the general mechanism References: <877fn61rmy.fsf@grothesque.org> <87twq9uewl.fsf@grothesque.org> Message-ID: <87oaghuemt.fsf@grothesque.org> Here is the missing link [1]: https://mail.scipy.org/pipermail/scipy-dev/2015-September/020932.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 810 bytes Desc: not available URL: From abarnert at yahoo.com Sat Oct 3 00:23:48 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 2 Oct 2015 15:23:48 -0700 Subject: [Python-ideas] asyncio: factoring-out of the general mechanism In-Reply-To: <87twq9uewl.fsf@grothesque.org> References: <877fn61rmy.fsf@grothesque.org> <87twq9uewl.fsf@grothesque.org> Message-ID: On Oct 2, 2015, at 14:41, Christoph Groth wrote: > > Guido van Rossum wrote: > >> Well, what else did you have in mind? Remember that until we fix the GIL >> (which won't happen until Python 4) you won't get benefit from async >> programming unless you are overlapping with I/O, and (due to the >> limitations of select/poll/etc.) that is pretty much limited to network and >> IPC (disk I/O in particular cannot be overlapped in this way). > > I recently realized that asynchronous programming is also useful for > scientific computation: It allows to quite naturally express a complex > algorithm that at various stages launches background computation. > > A good application of this technique is adaptive numerical integration. > See [1] for an example script that uses asyncio. There, the > asynchronous master Python process performs the integration algorithm > (that is computationally cheap but has quite complex control flow), > while the integrand function is evaluated by worker processes. > > Now, this particular example uses concurrent.futures, but one could also > use MPI, scoop, or 0mq. This is what made me wonder about the > monolithic design of asyncio. Why would you want asyncio here? You explicitly need parallelism, and independent threads/processes rather than coroutines (with or without an event loop). And it seems like composable futures are exactly the abstraction you want. It seems like what you really need here is to extend (or make it easier for end users and third party libs to extend) concurrent.futures to work with 0mq, etc., so you can leave the controller code alone while swapping out the dispatch mechanism. From steve at pearwood.info Sat Oct 3 05:27:26 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 3 Oct 2015 13:27:26 +1000 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: <20151003032725.GL23642@ando.pearwood.info> On Fri, Oct 02, 2015 at 09:00:56AM -0700, Guido van Rossum wrote: > Bingo. IMO the exact same arguments that show why f'{x} {y}' is better than > '%s %s' % (x, y) applies to byte strings. It would be totally acceptable if > it only took bytes (and bytearray, and memoryview) and numbers (which we > can guarantee are rendered in ASCII only). As Chris A pointed out earlier, identifiers are not ASCII only. What are we to make of something like this? bf'{??????? + 1}' And don't say "re-write your code to only use ASCII variable names" :-) -- Steve From rosuav at gmail.com Sat Oct 3 05:33:12 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Oct 2015 13:33:12 +1000 Subject: [Python-ideas] Binary f-strings In-Reply-To: <20151003032725.GL23642@ando.pearwood.info> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <20151003032725.GL23642@ando.pearwood.info> Message-ID: On Sat, Oct 3, 2015 at 1:27 PM, Steven D'Aprano wrote: > On Fri, Oct 02, 2015 at 09:00:56AM -0700, Guido van Rossum wrote: >> Bingo. IMO the exact same arguments that show why f'{x} {y}' is better than >> '%s %s' % (x, y) applies to byte strings. It would be totally acceptable if >> it only took bytes (and bytearray, and memoryview) and numbers (which we >> can guarantee are rendered in ASCII only). > > As Chris A pointed out earlier, identifiers are not ASCII only. What are > we to make of something like this? > > bf'{??????? + 1}' > > And don't say "re-write your code to only use ASCII variable names" :-) It should be technically legal, btw; it's just going to look very odd. The check for ASCII-only has to be done _after_ the fracturing into strings and expressions. But I don't like how that reads. ChrisA From guido at python.org Sat Oct 3 05:44:23 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Oct 2015 20:44:23 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <20151003032725.GL23642@ando.pearwood.info> Message-ID: On Fri, Oct 2, 2015 at 8:33 PM, Chris Angelico wrote: > On Sat, Oct 3, 2015 at 1:27 PM, Steven D'Aprano > wrote: > > On Fri, Oct 02, 2015 at 09:00:56AM -0700, Guido van Rossum wrote: > >> Bingo. IMO the exact same arguments that show why f'{x} {y}' is better > than > >> '%s %s' % (x, y) applies to byte strings. It would be totally > acceptable if > >> it only took bytes (and bytearray, and memoryview) and numbers (which we > >> can guarantee are rendered in ASCII only). > > > > As Chris A pointed out earlier, identifiers are not ASCII only. What are > > we to make of something like this? > > > > bf'{??????? + 1}' > > > > And don't say "re-write your code to only use ASCII variable names" :-) > > It should be technically legal, btw; it's just going to look very odd. > The check for ASCII-only has to be done _after_ the fracturing into > strings and expressions. But I don't like how that reads. > I don't think this concern should be a showstopper. Honestly either approach sounds fine to me. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 3 16:05:19 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Oct 2015 00:05:19 +1000 Subject: [Python-ideas] secrets module -- secret.keeper? In-Reply-To: <560C263A.1010600@mgmiller.net> References: <560C263A.1010600@mgmiller.net> Message-ID: On 1 October 2015 at 04:13, Mike Miller wrote: > Somewhat related, there is a keyring module, the functionality of which I've > sometimes wished were part of the stdlib: > > https://pypi.python.org/pypi/keyring > > It supports the big three OSs. We've learned from experience that these kinds of module are better coupled to operating system update cycles than they are to Python language definition update cycles. However, a case could potentially be made for a pip-style bundling arrangement that provides the other benefits of stdlib inclusion (python-dev's technical endorsement, PSF's legal blessing), without incurring the costs (i.e. coupling to the language update cycle for feature additions). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 3 16:37:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Oct 2015 00:37:04 +1000 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: On 3 October 2015 at 02:00, Guido van Rossum wrote: > Bingo. IMO the exact same arguments that show why f'{x} {y}' is better than > '%s %s' % (x, y) applies to byte strings. It would be totally acceptable if > it only took bytes (and bytearray, and memoryview) and numbers (which we can > guarantee are rendered in ASCII only). Given that restriction, what if we dropped the format() call in the bytestring case, and instead always used printf-style formatting? That is: bf'{packet_id}{chat_id}{sender_pk}{nonce}' could be roughly equivalent to (with parens to help make the pieces clearer): (b'%b' % packet_id) + (b'%b' % chat_id) + (b'%b' % sender_pk) + (b'%b' % nonce) If a ":fmt" section is provided for the substitution field, it would replace the mod-format sequence for that section: bf'{number:0.2d} ===> b'%0.2d' % number With that approach, over time, printf-style formatting (aka mod-formatting) may come to just be known as bytes formatting (even though text strings also support it). Something else that's neat with this: you could use the struct module for more complex subsections of a binary protocol, while doing the higher level composition with bf-strings*: bf'{header}{struct.pack('<10sHHb', record)}{footer}' Cheers, Nick. * which I am now tempted to call Big Friendly Strings**, since I read a lot of Roald Dahl books as a kid :) ** this would further mean that normal f-strings are friendly strings in addition to being format strings ;) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Sat Oct 3 16:53:56 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 3 Oct 2015 23:53:56 +0900 Subject: [Python-ideas] Binary f-strings In-Reply-To: <20151003032725.GL23642@ando.pearwood.info> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <20151003032725.GL23642@ando.pearwood.info> Message-ID: <22031.60420.627606.122630@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > What are we to make of something like this? > > bf'{??????? + 1}' Greek authorship. What would you make of bf'{junban + 1}' and why is that better than bf'{?? + 1}' ? (I would guess that is a sort of Japanese approximation to your Greek.) I think it was Alex Martelli who argued very strongly at the time of PEP 263 that using native identifiers and even comments in your native language is a very risky practice (at least from management's POV ). I think that's still true, but it's clearly in consenting adults territory. From bondarenkoedik at gmail.com Sat Oct 3 17:05:26 2015 From: bondarenkoedik at gmail.com (Eduard Bondarenko) Date: Sat, 3 Oct 2015 18:05:26 +0300 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list Message-ID: Hello everyone, the main idea of this proposal is to create convenient, maybe more Pythonic way to remove all occurrences of a value from a Python list. Suppose we have list 'arr': arr = [1, 2, 3, 1] and we want to remove all 1 from this list. The most Pythonic way to do it is: arr[:] = (x for x in arr if x != 1) Looks good, at least for experienced developer, but for Python's newcomers solution will look like this: >>> while True: ... try: ... a.remove(1) ... except: ... break I was surprised that Python doesn't have easy way to remove all occurrences. Currently I am reading "Effective Python" book and I have encounter good idea that it's also important to have readable code for new or another-language developers. And to my mind current Pythonic 'remove all occurrences' is not readable code and does not give insight (at least at first glance) into what happens in the code. This way looks a little bit nicer. I think so.. a.remove(1, all = True) So, that's idea in brief. Bellow you can find description how to install and test patch. Patch is attached. Note: this is test patch, so some things may not be done in accordance with PEP 007. How to install patch: hg update 3.5 cd Objects patch < remove_all.patch make How to use patch: >>> arr = [1, 2, 3, 1] >>> arr.remove(1) >>> arr [2, 3, 1] >>> arr = [1, 2, 3, 1] >>> arr.remove(1, all = True) >>> arr [2, 3] >>> arr = [1, 2, 3, 1] >>> arr.remove(1, True) Traceback (most recent call last): File "", line 1, in TypeError: remove() takes one argument >>> arr.remove(1, al = True) Traceback (most recent call last): File "", line 1, in TypeError: 'al' is an invalid keyword argument for this function Many thanks for your attention! - Eduard -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: remove_all.patch Type: application/octet-stream Size: 2176 bytes Desc: not available URL: From steve.dower at python.org Sat Oct 3 18:20:20 2015 From: steve.dower at python.org (Steve Dower) Date: Sat, 3 Oct 2015 09:20:20 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: "Something else that's neat with this: you could use the struct module for more complex subsections of a binary protocol" Personally, if binary f-strings did struct packing by default, I'd want to use them all the time. bf'{header}{record:<10sHHb}{footer}' Practically, if they aren't equivalent to removing the b and encoding the resulting f-string, I expect we'll regularly hit confusion and misunderstanding. $0.02 Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Nick Coghlan" Sent: ?10/?3/?2015 7:37 To: "Guido van Rossum" Cc: "Python-Ideas" Subject: Re: [Python-ideas] Binary f-strings On 3 October 2015 at 02:00, Guido van Rossum wrote: > Bingo. IMO the exact same arguments that show why f'{x} {y}' is better than > '%s %s' % (x, y) applies to byte strings. It would be totally acceptable if > it only took bytes (and bytearray, and memoryview) and numbers (which we can > guarantee are rendered in ASCII only). Given that restriction, what if we dropped the format() call in the bytestring case, and instead always used printf-style formatting? That is: bf'{packet_id}{chat_id}{sender_pk}{nonce}' could be roughly equivalent to (with parens to help make the pieces clearer): (b'%b' % packet_id) + (b'%b' % chat_id) + (b'%b' % sender_pk) + (b'%b' % nonce) If a ":fmt" section is provided for the substitution field, it would replace the mod-format sequence for that section: bf'{number:0.2d} ===> b'%0.2d' % number With that approach, over time, printf-style formatting (aka mod-formatting) may come to just be known as bytes formatting (even though text strings also support it). Something else that's neat with this: you could use the struct module for more complex subsections of a binary protocol, while doing the higher level composition with bf-strings*: bf'{header}{struct.pack('<10sHHb', record)}{footer}' Cheers, Nick. * which I am now tempted to call Big Friendly Strings**, since I read a lot of Roald Dahl books as a kid :) ** this would further mean that normal f-strings are friendly strings in addition to being format strings ;) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Sat Oct 3 21:18:06 2015 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 03 Oct 2015 15:18:06 -0400 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: <561029EE.8050006@trueblade.com> On 10/03/2015 12:20 PM, Steve Dower wrote: > "Something else that's neat with this: you could use the struct module > for more complex subsections of a binary protocol" > > Personally, if binary f-strings did struct packing by default, I'd want > to use them all the time. That appeals to me, too. There are a number of practical problems that would need to be worked out. We can argue those later :) I guess it comes down to: what would the commonest use case for fb-strings be? > Practically, if they aren't equivalent to removing the b and encoding > the resulting f-string, I expect we'll regularly hit confusion and > misunderstanding. This is one of my two big concerns. If we do something other than remove 'b' and encode, then we've got two similar looking things that have vastly different implementations. But maybe struct.pack or %-formatting are so compelling that it's worth breaking the equivalence. My other concern is non-ascii chars inside the braces in an fb-string. Eric. From ron3200 at gmail.com Sat Oct 3 22:27:20 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 3 Oct 2015 15:27:20 -0500 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> Message-ID: On 10/02/2015 02:56 PM, Andrew Barnert via Python-ideas wrote: >> It's not quite the same. The proposed ?( skips the arguments if >> the right hand side is None. >>> >>> None?(...) ---> None >>> >>> So you can't use that to supply a default directly. >>> >>> value = None?(default) # returns None (not useful here) > Again, why would you want that to return default? I guess it depends on what you think it should do. None is a type, NoneType to be exact, then if like int, or float, what kind of argument should it have? '' and empty containers come to mind. Just like float and int can take '1', and error if you give it something that can't be converted to a float or int. The None?(...) syntax looks like it should do a call. Just like it would do if you did int?(...). So both the return None and the return everything are special behaviors. Possibly a None result is a bit more correct. To tell the truth I think looking at some real examples from the library would be good. How can those be improved both in how they read and how they look, and how frequent are they? This whole line of discussion started out as a very tenuous suggestion/possibility. I even said it was going out on a limb, so I really wasn't expecting to defend it as if it was a complete idea. The concept I am thing of is doable, but not in the way I was thinking at first, but I also think it doesn't fit with python very well. Cheers, Ron From tjreedy at udel.edu Sat Oct 3 23:54:40 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 3 Oct 2015 17:54:40 -0400 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: Message-ID: On 10/3/2015 11:05 AM, Eduard Bondarenko wrote: > Hello everyone, > > the main idea of this proposal is to create convenient, maybe more > Pythonic way to remove all occurrences of a value from a Python list. This is a special case of in-place filtering. > Suppose we have list 'arr': > > arr = [1, 2, 3, 1] > > > and we want to remove all 1 from this list. > > The most Pythonic way to do it is: > > > arr[:] = (x for x in arr if x != 1) arr[:] = filter(lambda x: x != 1, x) > Looks good, at least for experienced developer, but for Python's > newcomers solution will look like this: > > >>> while True: > ... try: > ... a.remove(1) > ... except: > ... break > Python programmers really must learn either comprehensions or map and filter. They must also learn that repeatedly scanning a sequence should be avoided when possible. > I was surprised that Python doesn't have easy way to remove all occurrences. The two ways above are pretty easy. > Currently I am reading "Effective Python" book and I have encounter good > idea that it's also important to have readable code for new or > another-language developers. And to my mind current Pythonic 'remove all > occurrences' is not readable code and does not give insight (at least at > first glance) into what happens in the code. > > This way looks a little bit nicer. I think so.. > > a.remove(1, all = True) I would not mind this, but I don't know if there are enough use cases to justify an addition. -- Terry Jan Reedy From abarnert at yahoo.com Sun Oct 4 00:25:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 3 Oct 2015 15:25:47 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: On Oct 3, 2015, at 09:20, Steve Dower wrote: > > "Something else that's neat with this: you could use the struct module > for more complex subsections of a binary protocol" > > Personally, if binary f-strings did struct packing by default, I'd want to use them all the time. > > bf'{header}{record:<10sHHb}{footer}' I love that at first glance. But if the point of bf-strings (like the point of bytes.__mod__ and the other str-like stuff added back to bytes since 3.0) is for things like ascii-based, partly-human-readable protocols and formats, it's obviously important to do things like hex and octal, space- and zero-padding, etc., and if the format specifier always means struct, there's no way to do that. > Practically, if they aren't equivalent to removing the b and encoding the resulting f-string, I expect we'll regularly hit confusion and misunderstanding. But removing the b and encoding the resulting f-string is useless. For example: header = b'Spam' value = 42 lines.append(bf'{header}: {value}\r\n') This gives you b"b'Spam': 42\r\n". Can you imagine ever wanting that? The only way the feature makes sense is if it does something different. Nick's suggestion of having it do %-formatting makes sense. Yes, it means that {count:03} is an error and you need '{count:03d}', which is inconsistent with f-strings. But that seems like a much less serious problem than bytes formatting not being able to handle bytes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Sun Oct 4 00:31:26 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 03 Oct 2015 18:31:26 -0400 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: Message-ID: <5610573E.3070607@sdamon.com> On 10/3/2015 17:54, Terry Reedy wrote: > On 10/3/2015 11:05 AM, Eduard Bondarenko wrote: >> Hello everyone, >> >> the main idea of this proposal is to create convenient, maybe more >> Pythonic way to remove all occurrences of a value from a Python list. > > This is a special case of in-place filtering. > >> Suppose we have list 'arr': >> >> arr = [1, 2, 3, 1] >> >> >> and we want to remove all 1 from this list. >> >> The most Pythonic way to do it is: >> >> >> arr[:] = (x for x in arr if x != 1) > > arr[:] = filter(lambda x: x != 1, x) > >> Looks good, at least for experienced developer, but for Python's >> newcomers solution will look like this: >> >> >>> while True: >> ... try: >> ... a.remove(1) >> ... except: >> ... break >> > > Python programmers really must learn either comprehensions or map and > filter. They must also learn that repeatedly scanning a sequence > should be avoided when possible. > >> I was surprised that Python doesn't have easy way to remove all >> occurrences. > > The two ways above are pretty easy. > >> Currently I am reading "Effective Python" book and I have encounter good >> idea that it's also important to have readable code for new or >> another-language developers. And to my mind current Pythonic 'remove all >> occurrences' is not readable code and does not give insight (at least at >> first glance) into what happens in the code. >> >> This way looks a little bit nicer. I think so.. >> >> a.remove(1, all = True) > > I would not mind this, but I don't know if there are enough use cases > to justify an addition. > I am generally opposed to this. This is sugar for filtering on a predicate, with the predicate simply being predefined as `bar == foo`. There are a number of other common filtering use cases, should we include them all? Does this make python an easier language to use? In the new user case, does this teach, or encourage the new user to intuit, how any other filtering operations should be written? Would it not benefit the new user more to just have them learn listcomps, map, filter, and even while loops? From abarnert at yahoo.com Sun Oct 4 00:59:06 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 3 Oct 2015 15:59:06 -0700 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> Message-ID: On Oct 3, 2015, at 13:27, Ron Adam wrote: > > On 10/02/2015 02:56 PM, Andrew Barnert via Python-ideas wrote: >>> It's not quite the same. The proposed ?( skips the arguments if >>> the right hand side is None. >>>> >>>> None?(...) ---> None >>>> >>>> So you can't use that to supply a default directly. >>>> >>>> value = None?(default) # returns None (not useful here) >> Again, why would you want that to return default? > > I guess it depends on what you think it should do. None is a type, NoneType to be exact, then if like int, or float, what kind of argument should it have? None isn't a type. None _has_ a type, just like 42 does. And, given that, it does exactly what I'd expect: >>> 42() TypeError: 'int' object is not callable >>> None() TypeError: 'NoneType' object is not callable >>> type(42)() 0 >>> type(None)() None >>> 42('23') TypeError: 'int' object is not callable >>> None('23') TypeError: 'NoneType' object is not callable >>> type(42)('23') 23 >>> type(None)('23') TypeError: NoneType takes no arguments Calling int('23') makes sense as a conversion constructor; calling NoneType('23') doesn't. But, even if you insisted that it should mean something, it would make far more sense for it to return None than '23'. After all, it's a constructor, it ought to construct an object of its type. But that still wouldn't mean that None should be callable just because NoneType is, any more than it means that 42 should be callable just because int is. Meanwhile, you still haven't answered what "return the arguments" should actually mean, given that in Python, "the arguments" are a tuple of positional arguments plus a dict of keyword arguments. What would be the point in returning that? In a language where a 1-tuple is the same thing as its element, like the ML family or Haskell, and where there are no keyword arguments, something like this might be sensible. But in Python, it's not. Unless you have a way to make sense of it that I haven't thought of--which is why I'm asking. > '' and empty containers come to mind. Just like float and int can take '1', and error if you give it something that can't be converted to a float or int. > > The None?(...) syntax looks like it should do a call. Just like it would do if you did int?(...). Again, int is a type, and None isn't. Callability is a special property that only certain types--functions, methods, types, and things that want to act like one of the above--have. None is not a function, method, or type, any more than 42 is. Second, the whole reason spam?(?) has been proposed in the first place is that None(?) obviously doesn't make sense, so we want a way to avoid evaluating it if spam is None. In a language where everything is callable, there's no reason to even talk about ?(). > So both the return None and the return everything are special behaviors. Possibly a None result is a bit more correct. Making it return None isn't more correct, at least not for Python. It's definitely more correct for a language like SmallTalk with an "active null", which should propagate through almost all expressions by making them evaluate to None rather than raising. In such a language, for None(?) to do anything but return None would be inconsistent. But in a language where None doesn't propagate through any expressions, and instead usually raises a TypeError, for None(?) to return None would be highly inconsistent. And if you're suggesting that we should replace None with a SmallTalk-style active null, that s a much bigger change, which would have to change a lot more than NoneType.__call__, in a way that would radically affect the semantics of millions of APIs, lines of code, etc. > To tell the truth I think looking at some real examples from the library would be good. How can those be improved both in how they read and how they look, and how frequent are they? People have given real examples (from their own code, not the stdlib) for ?() as defined in the proposal. If you want examples for your different proposal, I think you're the one who needs to give them. (And I agree they would be worthwhile.) > This whole line of discussion started out as a very tenuous suggestion/possibility. I even said it was going out on a limb, so I really wasn't expecting to defend it as if it was a complete idea. > > The concept I am thing of is doable, but not in the way I was thinking at first, but I also think it doesn't fit with python very well. I think either it isn't doable, or it fits into Python even more poorly than you think. But I'm just one guy, who could easily be wrong. If you think otherwise, I would be interested in seeing what you can come up with. Even if it ends up not being a proposal you'd want to back, it would probably give insights to half the people involved in the larger discussion on null coalescing and null conditionals. And if it turns out that I'm right and you can't make it work, convincing yourself of that might lead to useful insights into the larger discussion that you could share here. After all, it seems like we're kind of stuck on conflicting incomplete intuitions here, and just arguing those back and forth isn't getting us very far. From abarnert at yahoo.com Sun Oct 4 01:32:04 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 3 Oct 2015 16:32:04 -0700 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: Message-ID: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> On Oct 3, 2015, at 14:54, Terry Reedy wrote: > >> On 10/3/2015 11:05 AM, Eduard Bondarenko wrote: >> Hello everyone, >> >> the main idea of this proposal is to create convenient, maybe more >> Pythonic way to remove all occurrences of a value from a Python list. > > This is a special case of in-place filtering. > >> Suppose we have list 'arr': >> >> arr = [1, 2, 3, 1] >> >> >> and we want to remove all 1 from this list. >> >> The most Pythonic way to do it is: >> >> >> arr[:] = (x for x in arr if x != 1) > > arr[:] = filter(lambda x: x != 1, x) I don't think this is more Pythonic. You've wrapped a perfectly good expression in an unnecessary function just so you can unnecessarily use a higher-order function. (Of course, by the same token, if you already have a perfectly good function lying around, using filter makes more sense than wrapping it up in an unnecessary call expression just so you can avoid a higher-order function. And there are cases where is unclear which is more appropriate--e.g., if what you have lying around is type or instance with a bound or unbound method, or something you can call partial on, is that really better than wrapping it in an expression? But this isn't either of those cases.) I'd have no problem with a novice who didn't yet know comprehensions writing this, but I wouldn't want to teach it as a better alternative to comprehensions in cases where it's not actually better. >> Looks good, at least for experienced developer, but for Python's >> newcomers solution will look like this: >> >> >>> while True: >> ... try: >> ... a.remove(1) >> ... except: >> ... break > > Python programmers really must learn either comprehensions or map and filter. They must also learn that repeatedly scanning a sequence should be avoided when possible. Agreed. In my experience, most novices who ask about code like this, once you explain to them that remove() has to keep repeatedly scanning from the start, immediately see why that's bad and ask how you can avoid doing that. That's a perfect opportunity to teach them about comprehensions while they're looking for exactly what comprehensions can do. There are some exceptions, but those aren't novices, they're intermediate-experienced C devs who insist that the following is the "simplest" code and must be the fastest or Python is broken (even though it's not necessarily fastest even in C, largely because memmove is so much faster than element by element move): idx = skip = 0 length = len(arr) while idx+skip < length: if arr[idx+skip] == element_to_remove: skip += 1 else: arr[idx] = arr[idx+skip] idx += 1 del arr[idx+1:] >> Currently I am reading "Effective Python" book and I have encounter good >> idea that it's also important to have readable code for new or >> another-language developers. And to my mind current Pythonic 'remove all >> occurrences' is not readable code and does not give insight (at least at >> first glance) into what happens in the code. It's readable to anyone who understands comprehensions and slice assignment. And both of those are such fundamental concepts that, if you don't understand either of them yet at all, your intuitions aren't very good yet. But notice that there's nothing stopping you from wrapping this up in a function, or adding whitespace or comments to help yourself work through it: def remove_all(arr, value): """Remove all instances of value from arr""" arr[:] = (x for x in arr # keep every element if x != value) # that doesn't equal value And now, everywhere you use it looks like this: remove_all(arr, 1) And it's hard to imagine anything more readable. And, even if remove_all isn't the kind of function an experienced developer would write, learning how to factor out the tricky bits into documentable and testable functions is one of the most useful skills for any developer in any language. From rob.cliffe at btinternet.com Sun Oct 4 01:14:15 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 4 Oct 2015 00:14:15 +0100 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: Message-ID: <56106147.8000704@btinternet.com> On 03/10/2015 22:54, Terry Reedy wrote: > On 10/3/2015 11:05 AM, Eduard Bondarenko wrote: >> Hello everyone, >> >> the main idea of this proposal is to create convenient, maybe more >> Pythonic way to remove all occurrences of a value from a Python list. > > This is a special case of in-place filtering. > >> Suppose we have list 'arr': >> >> arr = [1, 2, 3, 1] >> >> >> and we want to remove all 1 from this list. >> >> The most Pythonic way to do it is: >> >> >> arr[:] = (x for x in arr if x != 1) > > arr[:] = filter(lambda x: x != 1, x) I think you mean arr[:] = filter(lambda x: x!=1, arr) which suggests that using filter/map is not quite as easy as its advocates believe. :-) > >> >> This way looks a little bit nicer. I think so.. >> >> a.remove(1, all = True) > Other things that might be useful (sorry, I have no use cases in mind): a.remove(1, count) # maximum number of removals, analogous to aString.replace(old, new, count) a remove-like function that does not raise an error if the item is not present a remove-like function that returns the mutated list (like sorted(), as opposed to list.sort() ) It's not obvious to me how to design good API(s) to do some/all of this. Rob Cliffe From ron3200 at gmail.com Sun Oct 4 03:58:19 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 3 Oct 2015 20:58:19 -0500 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: References: <56099C6F.90700@oddbird.net> <36AB4531-96BD-4D22-A957-B2199BA7912E@stufft.io> <85oagm2saa.fsf@benfinney.id.au> <5609AB62.5040503@oddbird.net> <20150929133542.4d04f6dd@anarchist.wooz.org> <560C87A3.1020507@mrabarnett.plus.com> <5CF3DF17-D9FD-46B6-B74B-A161855D8B04@yahoo.com> <1443733946.502049.399030513.749E8ACE@webmail.messagingengine.com> <1443790486.723247.399552329.2D51881A@webmail.messagingengine.com> Message-ID: On 10/03/2015 05:59 PM, Andrew Barnert via Python-ideas wrote: > Meanwhile, you still haven't answered what "return the arguments" > should actually mean, given that in Python, "the arguments" are a > tuple of positional arguments plus a dict of keyword arguments. The simplest example is if a function has 5 arguments, then it will return those in the order they are in the signature. >>> def foo(a, b, c, x=1, y=2): ... return a, b, c, x, y >>> (a, b, c, x, y) = foo(1, 2, 3, 4, 5) >>> a, b, c, x, y (1, 2, 3, 4, 5) >>> a = foo(1, 2, 3, 4, 5) >>> a (1, 2, 3, 4, 5) But I think that may not be as useful as I was thinking. Only a few functions return the same number of arguments as they take. It would actually be more applicable to filters, where a function is applied to each item, and None might be the default to do nothing with that item. >>> This whole line of discussion started out as a very tenuous >>> suggestion/possibility. I even said it was going out on a limb, >>> so I really wasn't expecting to defend it as if it was a >>> complete idea. >>> >>> The concept I am thing of is doable, but not in the way I was >>> thinking at first, but I also think it doesn't fit with python >>> very well. > I think either it isn't doable, or it fits into Python even more > poorly than you think. But I'm just one guy, who could easily be > wrong. If you think otherwise, I would be interested in seeing what > you can come up with. Even if it ends up not being a proposal you'd > want to back, it would probably give insights to half the people > involved in the larger discussion on null coalescing and null > conditionals. And if it turns out that I'm right and you can't make > it work, convincing yourself of that might lead to useful insights > into the larger discussion that you could share here. After all, it > seems like we're kind of stuck on conflicting incomplete intuitions > here, and just arguing those back and forth isn't getting us very > far. That's exactly what we are doing. ;-) I doubt I can do better than what's already suggested at this point. Cheers, Ron From steve at pearwood.info Sun Oct 4 15:22:03 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Oct 2015 00:22:03 +1100 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: Message-ID: <20151004132203.GN23642@ando.pearwood.info> On Sat, Oct 03, 2015 at 06:05:26PM +0300, Eduard Bondarenko wrote: > Hello everyone, > > the main idea of this proposal is to create convenient, maybe more Pythonic > way to remove all occurrences of a value from a Python list. What's so special about removing all occurances from a list? How often do you find yourself doing such a thing? These aren't rhetorical questions. They're questions that need to be answered before your proposal can be turned into a new feature. If the answer is "not very special, and only rarely" that won't necessarily rule out the change, but it will make it harder to convince that it is desirable. > Suppose we have list 'arr': [...] > Looks good, at least for experienced developer, but for Python's newcomers > solution will look like this: > > >>> while True: > ... try: > ... a.remove(1) > ... except: > ... break Part of the process of learning to be a programmer is learning to avoid awful code like the above and instead learn general purpose processing techniques like the list comp. In my experience, e beginner is more likely to come up with this: while 1 in arr: arr.remove(1) since it is far more straight-forward than your version, and involves fewer concepts ("what's try...except do?"). > I was surprised that Python doesn't have easy way to remove all > occurrences. That depends on what you mean by "easy". Or "obvious". I would consider the list comp to be both. But of course, I don't expect beginners to see things the same way. > Currently I am reading "Effective Python" book and I have > encounter good idea that it's also important to have readable code for > new or another-language developers. And to my mind current Pythonic > 'remove all occurrences' is not readable code and does not give > insight (at least at first glance) into what happens in the code. I disagree. The list comp *does* give insight into what happens in the code: you iterate over the existing elements of the list, collecting the ones which don't equal 1, and then save them back into the list. Whereas your suggestion: > This way looks a little bit nicer. I think so.. > > a.remove(1, all = True) is just a mysterious method call. What insight does it give? How does it work? There is no hint, no clue. It might as well be magic. On the other hand, once the programmer can reason about the task well enough to write the list comprehension, they can easily extend it to slightly different tasks: # Skip the first 8 items, then remove numbers less than 5: arr[9:] = [x for x in arr[9:] if x >= 5] Your remove(all=True) method cannot be extended and teaches the programmer nothing except how to solve this one problem. -- Steve From steve.dower at python.org Sun Oct 4 17:34:22 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 4 Oct 2015 08:34:22 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: Maybe we could spell it {spam!p:<10sHHb}? Top-posted from my Windows Phone -----Original Message----- From: "Andrew Barnert" Sent: ?10/?3/?2015 15:31 To: "Steve Dower" Cc: "Nick Coghlan" ; "Guido van Rossum" ; "Python-Ideas" Subject: Re: [Python-ideas] Binary f-strings On Oct 3, 2015, at 09:20, Steve Dower wrote: "Something else that's neat with this: you could use the struct module for more complex subsections of a binary protocol" Personally, if binary f-strings did struct packing by default, I'd want to use them all the time. bf'{header}{record:<10sHHb}{footer}' I love that at first glance. But if the point of bf-strings (like the point of bytes.__mod__ and the other str-like stuff added back to bytes since 3.0) is for things like ascii-based, partly-human-readable protocols and formats, it's obviously important to do things like hex and octal, space- and zero-padding, etc., and if the format specifier always means struct, there's no way to do that. Practically, if they aren't equivalent to removing the b and encoding the resulting f-string, I expect we'll regularly hit confusion and misunderstanding. But removing the b and encoding the resulting f-string is useless. For example: header = b'Spam' value = 42 lines.append(bf'{header}: {value}\r\n') This gives you b"b'Spam': 42\r\n". Can you imagine ever wanting that? The only way the feature makes sense is if it does something different. Nick's suggestion of having it do %-formatting makes sense. Yes, it means that {count:03} is an error and you need '{count:03d}', which is inconsistent with f-strings. But that seems like a much less serious problem than bytes formatting not being able to handle bytes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Mon Oct 5 08:28:49 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 05 Oct 2015 08:28:49 +0200 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> Message-ID: <561218A1.6030100@mail.de> On 04.10.2015 01:32, Andrew Barnert via Python-ideas wrote: > And now, everywhere you use it looks like this: > > remove_all(arr, 1) > > And it's hard to imagine anything more readable. arr.remove_all(1) > And, even if remove_all isn't the kind of function an experienced developer would write, learning how to factor out the tricky bits into documentable and testable functions is one of the most useful skills for any developer in any language.# True. Btw. the same is true for Python core devs. This said, I would appreciate the method 'remove_all' provided by the stdlib. ;-) Best, Sven From srkunze at mail.de Mon Oct 5 08:59:39 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Mon, 05 Oct 2015 08:59:39 +0200 Subject: [Python-ideas] Binary f-strings In-Reply-To: <561029EE.8050006@trueblade.com> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <561029EE.8050006@trueblade.com> Message-ID: <56121FDB.3050401@mail.de> On 03.10.2015 21:18, Eric V. Smith wrote: > On 10/03/2015 12:20 PM, Steve Dower wrote: >> "Something else that's neat with this: you could use the struct module >> for more complex subsections of a binary protocol" >> >> Personally, if binary f-strings did struct packing by default, I'd want >> to use them all the time. > That appeals to me, too. There are a number of practical problems that > would need to be worked out. We can argue those later :) I think that's were I reach the limit of my "binary" experience in Python. But if people (here Steve) found it useful, why not? If there are problems that cannot be solved easily, we can do a V1 and later a V2 that includes the struct packing. > I guess it comes down to: what would the commonest use case for > fb-strings be? To me, it's the same as for all f-strings: the bloody easy string concatenation of easily distinguishable parts. I even have to admit I thought they were called *format strings* because they *give format/structure* to the resulting string. Well, now I know better (format refers to the formatting of the input expressions) but the analogy is still in my mind. > My other concern is non-ascii chars inside the braces in an fb-string. What's wrong with them? Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Mon Oct 5 10:46:40 2015 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 5 Oct 2015 04:46:40 -0400 Subject: [Python-ideas] Binary f-strings In-Reply-To: <56121FDB.3050401@mail.de> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <561029EE.8050006@trueblade.com> <56121FDB.3050401@mail.de> Message-ID: <561238F0.20200@trueblade.com> >> My other concern is non-ascii chars inside the braces in an fb-string. > > What's wrong with them? It has to do with the order of processing we defined for regular f-strings. I'm still working through it to see what the implications are. Eric. From random832 at fastmail.com Mon Oct 5 16:06:07 2015 From: random832 at fastmail.com (Random832) Date: Mon, 05 Oct 2015 10:06:07 -0400 Subject: [Python-ideas] PEP 505 (None coalescing operators) thoughts In-Reply-To: <1082717728.2280152.1443469633455.JavaMail.yahoo@mail.yahoo.com> References: <1082717728.2280152.1443469633455.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1444053967.1647566.401641825.08F2130B@webmail.messagingengine.com> On Mon, Sep 28, 2015, at 15:47, Andrew Barnert via Python-ideas wrote: > spam?.eggs.cheese becomes this pseudo-AST (I've skipped the loads and > maybe some other stuff): > > Expr( > value=Attribute( > value=Attribute( > value=Name(id='spam'), attr='eggs', uptalk=True), > attr='cheese', uptalk=False)) > ? which is then compiled as this pseudo-bytecode: > > LOAD_NAME 'spam' > DUP_TOP > POP_JUMP_IF_NONE :label > LOAD_ATTR 'eggs' > LOAD_ATTR 'cheese' > :label To put this in more concrete terms... What pseudo-AST does (spam?.eggs).cheese end up as, if the notion that uptalk must not escape parentheses is accepted? The pseudo-bytecode is obvious: LOAD_NAME 'spam' JUMP_IF_NONE :label LOAD_ATTR 'eggs' :label LOAD_ATTR 'cheese' [If we're going to define a new opcode, might as well be one that doesn't require a dup] The AST, though, not so much. spam.eggs.cheese and (spam.eggs).cheese are identical: Expr(Attribute(Attribute(Name('spam'), 'eggs'), 'cheese')) And you gave this for the non-parenthesized spam?.eggs.cheese Expr(Attribute(Attribute(Name('spam'), 'eggs', True), 'cheese', False)) > I suppose the reference documentation wording is also important here, to > explain that an uptalked attributeref or subscription short-circuits the > whole primary. Primary's not the right term here, because a parenthesized expression is also a primary. And so is the first three (first two, first four) terms of a five-dot expression. In the grammar the thing we want is called atom_expr. Why *is* the BNF in the syntax documentation different from the BNF in the grammar anyway? From tjreedy at udel.edu Mon Oct 5 22:18:46 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 5 Oct 2015 16:18:46 -0400 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: <561218A1.6030100@mail.de> References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> Message-ID: On 10/5/2015 2:28 AM, Sven R. Kunze wrote: > On 04.10.2015 01:32, Andrew Barnert via Python-ideas wrote: >> And now, everywhere you use it looks like this: >> >> remove_all(arr, 1) >> >> And it's hard to imagine anything more readable. > > arr.remove_all(1) > >> And, even if remove_all isn't the kind of function an experienced >> developer would write, learning how to factor out the tricky bits into >> documentable and testable functions is one of the most useful skills >> for any developer in any language.# > > True. > > > Btw. the same is true for Python core devs. This said, I would > appreciate the method 'remove_all' provided by the stdlib. ;-) The problem with methods is that they only work with one class. A list.removeall would only remove things equal to a specific item from a list (and presumably in place). We already have both a generic filter function and syntax that will remove all items from any iterable that meet any condition. The stream can be fed into any other function that accept an iterable. -- Terry Jan Reedy From tjreedy at udel.edu Mon Oct 5 23:25:24 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 5 Oct 2015 17:25:24 -0400 Subject: [Python-ideas] Non-English names in the turtle module (with Spanish example) In-Reply-To: References: Message-ID: On 9/16/2015 6:28 PM, Al Sweigart wrote: > I've created a prototype for how we could add foreign language names to > the turtle.py module Before going very far with turtle.py, you should be aware that the current turtle.py, writen by Gregor Lingl, is a replacement for the first version. It has two problems. First, it was never properly contributed to Python. Lingl did not sign the contributor agreement before Martin von Loewis committed it, and has not done so subsequently when requested. It has its own copyright and proprietary license, which pydev has violated. Second, the original was written in tkinter, while Gregor's version is writen in his custom graphics language, which in turn is implemented in tkinter. Gregor planned to also implement his new intermediate layer in some other framework, though he never did. I personally will not touch the current turtle.py. I would rather revert to the original tkinter version, fix that, and move forward. But I am currently busy enough with IDLE. > and erase the language barrier for non-English schoolkids. The third problem is that this is not a a priority for any core developer. Turtle itself is not anyone's priority. > The Tortuga module has the same functionality as turtle, I presume > You can test it out by running "pip install tortuga" > > https://pypi.python.org/pypi/Tortuga You should keep this as a pypi project. Future turtle.py could be changed to load external dictionaries without being monkey patched, but even that would do nothing for existing installations. I feel strongly that language packs not be put in the stdlib. Each dictionary has to be checked for accuracy and suitability for kids. This is not a job for pydev. > Since Python 2 doesn't have simpledialog, Since python-ideas is for possible new features, and new features only go in future 3.x releases, code discussed here should be Python 3. > Check out the diff between Tortuga and turtle.py here: > https://www.diffchecker.com/2xmbrkhk > > This file can be easily adapted to support multiple programming languages. I think only one language should be loaded at a time. With 50 languages, it is possible that the same 'word' (sequence of letters) might be used by different languages to translate different turtle words. Then the last language loaded would win. > Thoughts? Suggestions? Good idea. Go ahead and do it -- on pypi. -- Terry Jan Reedy From mal at egenix.com Tue Oct 6 13:24:08 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 6 Oct 2015 13:24:08 +0200 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> Message-ID: <5613AF58.6070802@egenix.com> If you have more than a few values to remove, it's often faster to create a new list, since each removal will require a copy operation of all trailing items and (every now and then) a realloc of the list object to free up the unused space: new_arr = [x for x in arr if x != 1] On 05.10.2015 22:18, Terry Reedy wrote: > On 10/5/2015 2:28 AM, Sven R. Kunze wrote: >> On 04.10.2015 01:32, Andrew Barnert via Python-ideas wrote: >>> And now, everywhere you use it looks like this: >>> >>> remove_all(arr, 1) >>> >>> And it's hard to imagine anything more readable. >> >> arr.remove_all(1) >> >>> And, even if remove_all isn't the kind of function an experienced >>> developer would write, learning how to factor out the tricky bits into >>> documentable and testable functions is one of the most useful skills >>> for any developer in any language.# >> >> True. >> >> >> Btw. the same is true for Python core devs. This said, I would >> appreciate the method 'remove_all' provided by the stdlib. ;-) > > The problem with methods is that they only work with one class. A list.removeall would only remove > things equal to a specific item from a list (and presumably in place). We already have both a > generic filter function and syntax that will remove all items from any iterable that meet any > condition. The stream can be fed into any other function that accept an iterable. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From steve at pearwood.info Tue Oct 6 18:00:24 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Oct 2015 03:00:24 +1100 Subject: [Python-ideas] PEP 506 (secrets module) and token functions In-Reply-To: <20150926130715.GG23642@ando.pearwood.info> References: <20150919181612.GT31152@ando.pearwood.info> <20150926130715.GG23642@ando.pearwood.info> Message-ID: <20151006160024.GA28222@ando.pearwood.info> Hi all, An updated version of PEP 506 is now available: https://www.python.org/dev/peps/pep-0506/ If there are no major objections, I intend to take it to python-dev in a day or two for discussion and a ruling. Thank you to everyone who contributed to the discussion. -- Steve From emilrosendahlpetersen at outlook.com Tue Oct 6 21:25:00 2015 From: emilrosendahlpetersen at outlook.com (Emil Rosendahl Petersen) Date: Tue, 6 Oct 2015 12:25:00 -0700 Subject: [Python-ideas] string.replace should accept a list as a first argument Message-ID: I think string.replace should be changed accept a list as a first argument. That way, if I had this string: "There are a lot of undesirable people in this filthy world" Then I could do this, replace(['undesirable', 'filthy'], ''), in case that's what I wanted to do. Now, string.replace doesn't accept a list as its first argument, and complains about implicit conversion. Is there any great obstacle to just having the function loop over that list, calling itself in case we get a list argument instead of a str? Doesn't that seem like the more obvious behaviour? To me the results of running the above code should be unsurprising, if this change was implemented: "there are a lot of people in this world". / Emil Petersen From srkunze at mail.de Tue Oct 6 18:29:54 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 06 Oct 2015 18:29:54 +0200 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: <5613AF58.6070802@egenix.com> References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> <5613AF58.6070802@egenix.com> Message-ID: <5613F702.2000502@mail.de> On 06.10.2015 13:24, M.-A. Lemburg wrote: > If you have more than a few values to remove, it's often > faster to create a new list, since each removal will > require a copy operation of all trailing items and (every > now and then) a realloc of the list object to free > up the unused space: > > new_arr = [x > for x in arr > if x != 1] > I think that's a technical detail which can be sorted out in C somehow more efficiently. But the proposal is not about performance rather than readability and maintainability. > On 05.10.2015 22:18, Terry Reedy wrote: >> The problem with methods is that they only work with one class. A list.removeall would only remove >> things equal to a specific item from a list (and presumably in place). We already have both a >> generic filter function and syntax that will remove all items from any iterable that meet any >> condition. The stream can be fed into any other function that accept an iterable. remove also removes in-place but it's not overly useful when you know or unsure about whether you need to remove an item multiple times. remove_all would basically complement remove as a more generic alternative. Btw. one could also think of an additional generalization for this that basically takes n arguments which are then removed altogether from the list in question: a = [1,2,3,5,4,2,3,4,1,2] a.remove_all(1,4,2) a == [3,5,3] Just thinking, and I might remember some time where I would have found it useful. Not sure. Best, Sven From mal at egenix.com Tue Oct 6 18:34:16 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 6 Oct 2015 18:34:16 +0200 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: Message-ID: <5613F808.1050702@egenix.com> On 06.10.2015 21:25, Emil Rosendahl Petersen wrote: > I think string.replace should be changed accept a list as a first > argument. > > That way, if I had this string: > > "There are a lot of undesirable people in this filthy world" > > Then I could do this, replace(['undesirable', 'filthy'], ''), in case > that's what I wanted to do. > > Now, string.replace doesn't accept a list as its first argument, and > complains about implicit conversion. > > Is there any great obstacle to just having the function loop over that > list, calling itself in case we get a list argument instead of a str? > > Doesn't that seem like the more obvious behaviour? To me the results of > running the above code should be unsurprising, if this change was > implemented: "there are a lot of people in this world". I think the "one obvious way" of doing a multi-replace is to use the re module, since implementing this efficiently is non-trivial. String methods are meant to be basic (high performance) operations. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rymg19 at gmail.com Tue Oct 6 18:37:20 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 6 Oct 2015 11:37:20 -0500 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: Message-ID: import re result = re.sub('undesirable|filthy', '', 'There are a lot of undesirable people in this filthy world') print(result) # There are a lot of people in this world On Tue, Oct 6, 2015 at 2:25 PM, Emil Rosendahl Petersen < emilrosendahlpetersen at outlook.com> wrote: > I think string.replace should be changed accept a list as a first > argument. > > That way, if I had this string: > > "There are a lot of undesirable people in this filthy world" > > Then I could do this, replace(['undesirable', 'filthy'], ''), in case > that's what I wanted to do. > > Now, string.replace doesn't accept a list as its first argument, and > complains about implicit conversion. > > Is there any great obstacle to just having the function loop over that > list, calling itself in case we get a list argument instead of a str? > > Doesn't that seem like the more obvious behaviour? To me the results of > running the above code should be unsurprising, if this change was > implemented: "there are a lot of people in this world". > > / Emil Petersen > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Oct 6 19:03:38 2015 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 6 Oct 2015 18:03:38 +0100 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: Message-ID: <5613FEEA.1040703@mrabarnett.plus.com> On 2015-10-06 20:25, Emil Rosendahl Petersen wrote: > I think string.replace should be changed accept a list as a first > argument. > > That way, if I had this string: > > "There are a lot of undesirable people in this filthy world" > > Then I could do this, replace(['undesirable', 'filthy'], ''), in case > that's what I wanted to do. > > Now, string.replace doesn't accept a list as its first argument, and > complains about implicit conversion. > > Is there any great obstacle to just having the function loop over that > list, calling itself in case we get a list argument instead of a str? > > Doesn't that seem like the more obvious behaviour? To me the results of > running the above code should be unsurprising, if this change was > implemented: "there are a lot of people in this world". > Looping over the list is the wrong way to do it because each pass might produce a string that leads to different matches for the subsequent passes. I think that the replacements should be done based on the earliest longest match. From srkunze at mail.de Tue Oct 6 19:35:03 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 06 Oct 2015 19:35:03 +0200 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: Message-ID: <56140647.4030501@mail.de> On 06.10.2015 21:25, Emil Rosendahl Petersen wrote: > I think string.replace should be changed accept a list as a first > argument. I think I can understand the sentiment here. However, I would prefer a str.replace_all method that would accept it. I cannot think of any place, where I would want to use either a str or a list of str. So, this could hide bugs. Speaking of "replace", sometimes, I would love to pass an "replace all of these with all of those" dict, which is then processed internally. I can remember two times where we needed to write some kind of for-loop; which actually might produce wrong results: convert_dict = { '1': '2', '2': '3', } original = '12345' for from_, to in convert_dict.items(): original = original.replace(from_, to) Real-world examples don't really have this issue so we accepted this kind of workaround for smaller scripts. But I would prefer a stdlib solution for this. Best, Sven From srkunze at mail.de Tue Oct 6 19:47:30 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 06 Oct 2015 19:47:30 +0200 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: Message-ID: <56140932.1090700@mail.de> On 06.10.2015 18:37, Ryan Gonzalez wrote: > import re > > result = re.sub('undesirable|filthy', '', 'There are a lot of > undesirable people in this filthy world') > print(result) # There are a lot of people in this world Yes, but(TM): The OP has a list, not a string. Yes, he could create one by '|'.join(['undesirable', 'filthy']) but that's like: "Hey, I have some structured data, let's create some applesauce and have a secondary parser re-create structured data from it." IMHO that feels wrong. Best, Sven From guido at python.org Tue Oct 6 20:26:11 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Oct 2015 11:26:11 -0700 Subject: [Python-ideas] PEP 506 (secrets module) and token functions In-Reply-To: <20151006160024.GA28222@ando.pearwood.info> References: <20150919181612.GT31152@ando.pearwood.info> <20150926130715.GG23642@ando.pearwood.info> <20151006160024.GA28222@ando.pearwood.info> Message-ID: This is already looking good. An additional advantage to having a new module (as opposed to changing random) is that it could easily be backported as a PyPI package, all the way back to Python 2.7. I do still think that having a concrete proposal for what should (initially) go into secrets.py would make for a more compelling PEP. On Tue, Oct 6, 2015 at 9:00 AM, Steven D'Aprano wrote: > Hi all, > > An updated version of PEP 506 is now available: > > https://www.python.org/dev/peps/pep-0506/ > > > If there are no major objections, I intend to take it to python-dev > in a day or two for discussion and a ruling. > > Thank you to everyone who contributed to the discussion. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From moiein2000 at gmail.com Tue Oct 6 21:40:55 2015 From: moiein2000 at gmail.com (Matthew Einhorn) Date: Tue, 6 Oct 2015 15:40:55 -0400 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: <56140647.4030501@mail.de> References: <56140647.4030501@mail.de> Message-ID: On Tue, Oct 6, 2015 at 1:35 PM, Sven R. Kunze wrote: > ... > Speaking of "replace", sometimes, I would love to pass an "replace all of > these with all of those" dict, which is then processed internally. I can > remember two times where we needed to write some kind of for-loop; which > actually might produce wrong results: > > convert_dict = { > '1': '2', > '2': '3', > } > original = '12345' > for from_, to in convert_dict.items(): > original = original.replace(from_, to) > Also, you can do: re.sub('|'.join(convert_dict.keys()), lambda x: convert_dict[x.group(0)], original) Which is probably better as it performs the match at once. But of course you have to know ahead of time what the keys of the dict may be as they need to be escaped etc. Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Oct 6 22:59:59 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 6 Oct 2015 13:59:59 -0700 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: Message-ID: On Oct 6, 2015, at 12:25, Emil Rosendahl Petersen wrote: > > I think string.replace should be changed accept a list as a first > argument. Just a list? If I call it with a tuple, or some other kind of iterable, should I get a TypeError telling me to pass a str or list? But of course str is also an iterable of str, which makes it ambiguous if you do otherwise. Python does have a few functions they can take either one Foo or multiple Food (and Foo may itself be iterable), but everywhere else it uses tuple as the special type, not list. > That way, if I had this string: > > "There are a lot of undesirable people in this filthy world" > > Then I could do this, replace(['undesirable', 'filthy'], ''), in case > that's what I wanted to do. Even without the multiple arguments, this is a very strange use of replace. For one thing, it leaves extra spaces behind. Also, I suspect your be tempted to use it similarly if you wanted to remove "filth", and the complain that it turned "filthy" into "y". I suspect that, in your actual motivating code, you want to either split the string, filter it, and rejoin it, or use regular expressions or a more complicated parser. > Now, string.replace doesn't accept a list as its first argument, and > complains about implicit conversion. > > Is there any great obstacle to just having the function loop over that > list, calling itself in case we get a list argument instead of a str? There are various inconsistent things that replace could do with multiple arguments, but I think this is the one you'd be least likely to want when they differ, and that would surprise people the most. Consider cases like removing "fily" and "th": should removing the "th" from "filthy" make it eligible to have the remaining "fily" replaced? Does it matter which order they're passed in the list? What if the replacement string isn't "" but "c"; does that mean "filcy" can be replaced? Of course as long as you think of a string as a sequence of words rather than a sequence of characters, you don't think of these issues?which is exactly why I think you should probably be splitting the string into words, so you don't have to. Or, if you really do want to do this character by character, you need to think through what you mean by how it affects order, greediness, etc. Often, the simplest way to express that is a regular expression, in which case, just do that. If you explicitly want something that's hard to express in a regexp, it's probably uncommon enough that you don't want it as a method on str, and want to have the logic in clear Python code for later reading. > Doesn't that seem like the more obvious behaviour? To me the results of > running the above code should be unsurprising, if this change was > implemented: "there are a lot of people in this world". > > / Emil Petersen > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Wed Oct 7 00:50:15 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Oct 2015 09:50:15 +1100 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: <5613F702.2000502@mail.de> References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> <5613AF58.6070802@egenix.com> <5613F702.2000502@mail.de> Message-ID: On Wed, Oct 7, 2015 at 3:29 AM, Sven R. Kunze wrote: > Btw. one could also think of an additional generalization for this that > basically takes n arguments which are then removed altogether from the list > in question: > > a = [1,2,3,5,4,2,3,4,1,2] > a.remove_all(1,4,2) > a == [3,5,3] > > Just thinking, and I might remember some time where I would have found it > useful. Not sure. > The more generalizations you offer, the more tempting the comprehension looks. It already allows as many generalizations as you like, because comprehensions are a fully-supported part of the language. ChrisA From srkunze at mail.de Wed Oct 7 01:21:10 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 07 Oct 2015 01:21:10 +0200 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> <5613AF58.6070802@egenix.com> <5613F702.2000502@mail.de> Message-ID: <56145766.8000103@mail.de> On 07.10.2015 00:50, Chris Angelico wrote: > On Wed, Oct 7, 2015 at 3:29 AM, Sven R. Kunze wrote: >> Btw. one could also think of an additional generalization for this that >> basically takes n arguments which are then removed altogether from the list >> in question: >> >> a = [1,2,3,5,4,2,3,4,1,2] >> a.remove_all(1,4,2) >> a == [3,5,3] >> >> Just thinking, and I might remember some time where I would have found it >> useful. Not sure. >> > The more generalizations you offer, the more tempting the > comprehension looks. To you, not to everybody. > It already allows as many generalizations as you > like, because comprehensions are a fully-supported part of the > language. You miss the point. Common use-cases deserve methods on their own. People will never stop asking for meaningfully named methods, even when pretending comprehensions are the ultimate answer to all questions concerning lists/sets/dicts, They simply are not. Which variant conveys the intent of the developer more clearly? a.remove_all(1,4,2) a[:] = [x for x in a if x not in {1,4,2}] I have to admit, the latter variant has certain appeal if you love special characters. Best, Sven From rosuav at gmail.com Wed Oct 7 02:01:10 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Oct 2015 11:01:10 +1100 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: <56145766.8000103@mail.de> References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> <5613AF58.6070802@egenix.com> <5613F702.2000502@mail.de> <56145766.8000103@mail.de> Message-ID: On Wed, Oct 7, 2015 at 10:21 AM, Sven R. Kunze wrote: > You miss the point. Common use-cases deserve methods on their own. People > will never stop asking for meaningfully named methods, even when pretending > comprehensions are the ultimate answer to all questions concerning > lists/sets/dicts, They simply are not. > > Which variant conveys the intent of the developer more clearly? > > a.remove_all(1,4,2) > a[:] = [x for x in a if x not in {1,4,2}] > Maybe, but how many other variants do you need? "remove all elements 2.7 (Sven R. Kunze's message of "Tue, 06 Oct 2015 19:47:30 +0200") References: <56140932.1090700@mail.de> Message-ID: "Sven R. Kunze" writes: > On 06.10.2015 18:37, Ryan Gonzalez wrote: > The OP has a list, not a string. Yes, he could create one by > '|'.join(['undesirable', 'filthy']) but that's like: > > "Hey, I have some structured data, let's create some applesauce and > have a secondary parser re-create structured data from it." IMHO that > feels wrong. Not sure how to get around that, other than by creating a general "structured regex" module, to build a compiled regex from an abstract regex syntax tree rather than a string. Which actually might not be the worst thing in the world. I think there are some Lisp dialects that have something like that. From rosuav at gmail.com Wed Oct 7 02:59:31 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Oct 2015 11:59:31 +1100 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: <56140932.1090700@mail.de> Message-ID: On Wed, Oct 7, 2015 at 11:49 AM, Random832 wrote: > Not sure how to get around that, other than by creating a general > "structured regex" module, to build a compiled regex from an abstract > regex syntax tree rather than a string. Which actually might not be the > worst thing in the world. I think there are some Lisp dialects that have > something like that. Ooh, I like this idea. Kinda like the difference between building up an SQL statement by escaping all your input, and using a parameterized statement. ChrisA From python at mrabarnett.plus.com Wed Oct 7 03:05:42 2015 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 7 Oct 2015 02:05:42 +0100 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: <56140932.1090700@mail.de> Message-ID: <56146FE6.2050309@mrabarnett.plus.com> On 2015-10-07 01:59, Chris Angelico wrote: > On Wed, Oct 7, 2015 at 11:49 AM, Random832 wrote: >> Not sure how to get around that, other than by creating a general >> "structured regex" module, to build a compiled regex from an abstract >> regex syntax tree rather than a string. Which actually might not be the >> worst thing in the world. I think there are some Lisp dialects that have >> something like that. > > Ooh, I like this idea. Kinda like the difference between building up > an SQL statement by escaping all your input, and using a parameterized > statement. > There's always this: https://pypi.python.org/pypi/regex Look at "Named lists". ;-) From emilrosendahlpetersen at outlook.com Wed Oct 7 06:06:40 2015 From: emilrosendahlpetersen at outlook.com (Emil Rosendahl Petersen) Date: Tue, 6 Oct 2015 21:06:40 -0700 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> <5613AF58.6070802@egenix.com> <5613F702.2000502@mail.de> <56145766.8000103@mail.de> Message-ID: On Wed, 2015-10-07 at 11:01 +1100, Chris Angelico wrote: > On Wed, Oct 7, 2015 at 10:21 AM, Sven R. Kunze wrote: > > You miss the point. Common use-cases deserve methods on their own. People > > will never stop asking for meaningfully named methods, even when pretending > > comprehensions are the ultimate answer to all questions concerning > > lists/sets/dicts, They simply are not. > > > > Which variant conveys the intent of the developer more clearly? > > > > a.remove_all(1,4,2) > > a[:] = [x for x in a if x not in {1,4,2}] > > > > Maybe, but how many other variants do you need? "remove all elements > 2.7 you create adds cognitive load to everyone who reads the docs for the > list object, and every new feature of a method adds cognitive load to > understanding that method. Is it worth it? How common *is* this case? > Is it really worth having the method? Comprehensions already exist, > and are already general enough to handle all the variants. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ Python is supposed to make use of duck-typing. Well... I say the standard library should apply that philosophy. If we are supposed to be a dynamic language, why can't we be like perl, and do list operations if we see lists, and scalar/single operations if we see single value types? It really comes down to "quack like a duck", in this case - Is it possible to loop over each thing in the given list, treat it like a string, and replace it with the string we apply to each iteration? I believe it should be. print(['One thing', 'another thing', 'each of these', 'on their own line']) Looking at that... isn't the intent very obvious? Why should python not respect that? "A long thing".replace("long ", '') >>> a thing "A plyable soft tube an undescribable color.".replace(['soft', 'plyable'], '').replace('undescribable', 'indescribable') Isn't the behaviour of this code - as it would be, if this worked - fairly obvious? Frankly to me it seems much more readable and step-by-step to me than a comparable comprehension. What actually physically happens in the code is easily apparently, unlike in a comprehension, at least until you take in its local scope variables. And lets face it, nested comprehensions can be rather messy. Since Python does not encourage tail calls, there should be a good and efficient way to apply operations everywhere. But maybe map is best? I haven't seen anyone in this thread present an alternative to changing the function - Perhaps some trick of a decorator or similar. But there is no obvious syntax, in my opinion, for accomplishing what I did with the nested .replace calls. From rosuav at gmail.com Wed Oct 7 03:11:40 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Oct 2015 12:11:40 +1100 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: <56146FE6.2050309@mrabarnett.plus.com> References: <56140932.1090700@mail.de> <56146FE6.2050309@mrabarnett.plus.com> Message-ID: On Wed, Oct 7, 2015 at 12:05 PM, MRAB wrote: >> Ooh, I like this idea. Kinda like the difference between building up >> an SQL statement by escaping all your input, and using a parameterized >> statement. >> > There's always this: > > https://pypi.python.org/pypi/regex > > Look at "Named lists". ;-) The time machine strikes again! ChrisA From steve at pearwood.info Wed Oct 7 03:30:07 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Oct 2015 12:30:07 +1100 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: <5613F808.1050702@egenix.com> References: <5613F808.1050702@egenix.com> Message-ID: <20151007013007.GC28222@ando.pearwood.info> On Tue, Oct 06, 2015 at 06:34:16PM +0200, M.-A. Lemburg wrote: > On 06.10.2015 21:25, Emil Rosendahl Petersen wrote: > > I think string.replace should be changed accept a list as a first > > argument. > > > > That way, if I had this string: > > > > "There are a lot of undesirable people in this filthy world" > > > > Then I could do this, replace(['undesirable', 'filthy'], ''), in case > > that's what I wanted to do. > > > > Now, string.replace doesn't accept a list as its first argument, and > > complains about implicit conversion. [Emil] > > Is there any great obstacle to just having the function loop over that > > list, calling itself in case we get a list argument instead of a str? Looping over each replacement item is the wrong solution. Think of the result when one of the search strings is a substring of the replacement: py> source = "I ate a chicken salad, and she had a ham sandwich." py> for term in ["ham", "turkey", "chicken", "spam"]: ... source = source.replace(term, "spam and cheese") ... py> print(source) I ate a spam and cheese and cheese salad, and she had a spam and cheese and cheese sandwich. You need to be a bit more careful about how to do the replacements. > > Doesn't that seem like the more obvious behaviour? To me the results of > > running the above code should be unsurprising, if this change was > > implemented: "there are a lot of people in this world". [MAL] > I think the "one obvious way" of doing a multi-replace is to > use the re module, since implementing this efficiently is > non-trivial. > > String methods are meant to be basic (high performance) > operations. A similar issue was discussed last month, in the context of str.split rather than replace, and I talked about the pitfalls of using the re module: https://mail.python.org/pipermail/python-ideas/2015-September/036586.html The implementation isn't hard, but it's just tricky enough that some people will get it wrong, and just useful enough that a helper function will be a good idea. The question is, should that helper function be a string method, in the standard library, or merely something that you add to your own projects? def replace_all(source, old, new, count=None): if isinstance(old, str): return source.replace(old, new, count) elif isinstance(old, tuple) regex = '|'.join(re.escape(s) for s in old) return new.join(re.split(regex, source, count)) -- Steve From steve at pearwood.info Wed Oct 7 04:02:04 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Oct 2015 13:02:04 +1100 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: References: <47AF8D14-EA4D-460C-B487-39DD222A8C09@yahoo.com> <561218A1.6030100@mail.de> <5613AF58.6070802@egenix.com> <5613F702.2000502@mail.de> <56145766.8000103@mail.de> Message-ID: <20151007020204.GD28222@ando.pearwood.info> On Tue, Oct 06, 2015 at 09:06:40PM -0700, Emil Rosendahl Petersen wrote: > print(['One thing', 'another thing', 'each of these', 'on their own > line']) > > Looking at that... isn't the intent very obvious? Why should python not > respect that? Yes, the intent is obvious, because I am a sentient human who can read the English text and guess -- I emphasis that is it just a guess -- what you want. Do you believe that Python should do the same thing that I did? Read the individual strings, analyse them as English text, and understand that because the final item says "on their own line" that is your intent. But what if I wrote this instead: print(['one string per line', 'three strings per line', 'everything on one line', 'nah just kidding', 'print using two equal-spaced columns']) What is my intention now? If *you* can't guess what I want, how can the interpreter guess? Do I want the following printed one number per line or all numbers on one line? Should I see the list delimiters? Should the output be formatted into multiple columns? How many columns? Should each number be left-justified, right-justified? Centered? print([1, 2, 3, 4, 5, 6, 7, 8, 998, 999]) Trying to have a programming language intuit the programmers *intent* is a fool's errand: it cannot be done successfully. Computers cannot do what we want, they can only do what we tell them to do. > "A long thing".replace("long ", '') > >>> a thing > > "A plyable soft tube an undescribable color.".replace(['soft', > 'plyable'], '').replace('undescribable', 'indescribable') > > Isn't the behaviour of this code - as it would be, if this worked - > fairly obvious? No, of course not. If you think it is obvious, you haven't thought about it in enough detail. What happens if the replacement strings overlap? What happens if the new string contains one or more of the old strings as substrings? Should the order of the old strings make a difference to the final result? Why a list? Isn't that likely to indicate a programming error? If replace() took multiple target substrings to be replaced, I have answers to those questions. But I don't know if those answers are the same as your answers. Maybe they are, maybe they're not. Who knows? - overlapping target strings shouldn't make a difference; - neither should the replacement string containing one or more of the targets; - or the order of the targets; - but a list probably means a programming error, I would prefer to require a tuple of substrings to be consistent with other string methods, and to avoid any questions of what happens with arbitrary iterables. -- Steve From steve at pearwood.info Wed Oct 7 04:07:45 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 7 Oct 2015 13:07:45 +1100 Subject: [Python-ideas] PEP 506 (secrets module) and token functions In-Reply-To: References: <20150919181612.GT31152@ando.pearwood.info> <20150926130715.GG23642@ando.pearwood.info> <20151006160024.GA28222@ando.pearwood.info> Message-ID: <20151007020745.GE28222@ando.pearwood.info> On Tue, Oct 06, 2015 at 11:26:11AM -0700, Guido van Rossum wrote: > This is already looking good. An additional advantage to having a new > module (as opposed to changing random) is that it could easily be > backported as a PyPI package, all the way back to Python 2.7. > > I do still think that having a concrete proposal for what should > (initially) go into secrets.py would make for a more compelling PEP. Thanks Guido. I'm not sure how much more concrete a proposal you are looking for. The PEP now lists a sample implementation. I've described it as "pseudo-code" only to indicate that it may be incomplete (e.g. missing some imports to make it work, lacking in error checking). It also lacks docstrings and tests, but otherwise I think it is most of the module. It's actually not very large, because most of the implementation is elsewhere (e.g. the random module). Is there something else the PEP should include? Any other requested functions? There have been a few functions suggested that were requested, e.g. password generation. https://www.python.org/dev/peps/pep-0506/ -- Steve From random832 at fastmail.com Wed Oct 7 04:14:54 2015 From: random832 at fastmail.com (Random832) Date: Tue, 06 Oct 2015 22:14:54 -0400 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: (Chris Angelico's message of "Wed, 7 Oct 2015 11:59:31 +1100") References: <56140932.1090700@mail.de> Message-ID: Chris Angelico writes: > On Wed, Oct 7, 2015 at 11:49 AM, Random832 wrote: >> Not sure how to get around that, other than by creating a general >> "structured regex" module, to build a compiled regex from an abstract >> regex syntax tree rather than a string. Which actually might not be the >> worst thing in the world. I think there are some Lisp dialects that have >> something like that. > > Ooh, I like this idea. Kinda like the difference between building up > an SQL statement by escaping all your input, and using a parameterized > statement. Not really. My idea was that no string in regex syntax ever exists, but instead something like "(?a|b|c)d*" becomes more like this: Sequence(Alternate("a","b","c"), Star("d")), which the regex engine could then use instead of parsing a string to build a state machine. All strings would be literal strings - "a\.b" would become "a.b" and "a.b" would become Sequence("a", Dot(), "b") Then you could just use Alternate(*lst). From rymg19 at gmail.com Wed Oct 7 04:19:35 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 06 Oct 2015 21:19:35 -0500 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: References: <56140932.1090700@mail.de> Message-ID: <3B4C7819-0C14-4737-8D43-DFFD6A7A666E@gmail.com> On October 6, 2015 9:14:54 PM CDT, Random832 wrote: >Chris Angelico writes: > >> On Wed, Oct 7, 2015 at 11:49 AM, Random832 >wrote: >>> Not sure how to get around that, other than by creating a general >>> "structured regex" module, to build a compiled regex from an >abstract >>> regex syntax tree rather than a string. Which actually might not be >the >>> worst thing in the world. I think there are some Lisp dialects that >have >>> something like that. >> >> Ooh, I like this idea. Kinda like the difference between building up >> an SQL statement by escaping all your input, and using a >parameterized >> statement. > >Not really. My idea was that no string in regex syntax ever exists, but >instead something like "(?a|b|c)d*" becomes more like this: >Sequence(Alternate("a","b","c"), Star("d")), which the regex engine >could then use instead of parsing a string to build a state machine. >All strings would be literal strings - "a\.b" would become "a.b" and >"a.b" would become Sequence("a", Dot(), "b") > >Then you could just use Alternate(*lst). Isn't that kind of like PyParsing with a DFA? ...and that's a really cool idea. Completely coincidentally, I've been working on a JIT-ted regex library in C. There's no parser yet, and I was planning on de-exposing the internal structures, but, after reading this, I'll probably just leave it exposed. >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From python at mrabarnett.plus.com Wed Oct 7 04:24:40 2015 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 7 Oct 2015 03:24:40 +0100 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: <20151007013007.GC28222@ando.pearwood.info> References: <5613F808.1050702@egenix.com> <20151007013007.GC28222@ando.pearwood.info> Message-ID: <56148268.3030908@mrabarnett.plus.com> On 2015-10-07 02:30, Steven D'Aprano wrote: > On Tue, Oct 06, 2015 at 06:34:16PM +0200, M.-A. Lemburg wrote: >> On 06.10.2015 21:25, Emil Rosendahl Petersen wrote: >> > I think string.replace should be changed accept a list as a first >> > argument. >> > >> > That way, if I had this string: >> > >> > "There are a lot of undesirable people in this filthy world" >> > >> > Then I could do this, replace(['undesirable', 'filthy'], ''), in case >> > that's what I wanted to do. >> > >> > Now, string.replace doesn't accept a list as its first argument, and >> > complains about implicit conversion. > > [Emil] >> > Is there any great obstacle to just having the function loop over that >> > list, calling itself in case we get a list argument instead of a str? > > Looping over each replacement item is the wrong solution. Think of > the result when one of the search strings is a substring of the > replacement: > > py> source = "I ate a chicken salad, and she had a ham sandwich." > py> for term in ["ham", "turkey", "chicken", "spam"]: > ... source = source.replace(term, "spam and cheese") > ... > py> print(source) > I ate a spam and cheese and cheese salad, and she had a spam and cheese > and cheese sandwich. > > > You need to be a bit more careful about how to do the replacements. > > >> > Doesn't that seem like the more obvious behaviour? To me the results of >> > running the above code should be unsurprising, if this change was >> > implemented: "there are a lot of people in this world". > > [MAL] >> I think the "one obvious way" of doing a multi-replace is to >> use the re module, since implementing this efficiently is >> non-trivial. >> >> String methods are meant to be basic (high performance) >> operations. > > A similar issue was discussed last month, in the context of str.split > rather than replace, and I talked about the pitfalls of using the re > module: > > https://mail.python.org/pipermail/python-ideas/2015-September/036586.html > > > The implementation isn't hard, but it's just tricky enough that some > people will get it wrong, and just useful enough that a helper function > will be a good idea. The question is, should that helper function be a > string method, in the standard library, or merely something that you add > to your own projects? > > def replace_all(source, old, new, count=None): > if isinstance(old, str): > return source.replace(old, new, count) > elif isinstance(old, tuple) > regex = '|'.join(re.escape(s) for s in old) > return new.join(re.split(regex, source, count)) > > Again, using the regex module, you can split on a named list, without having to worry about sorting or escaping the items. :-) From rymg19 at gmail.com Wed Oct 7 04:36:00 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 06 Oct 2015 21:36:00 -0500 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: <56148268.3030908@mrabarnett.plus.com> References: <5613F808.1050702@egenix.com> <20151007013007.GC28222@ando.pearwood.info> <56148268.3030908@mrabarnett.plus.com> Message-ID: <76387C69-5DF0-4913-B969-5EEE5D6EFC50@gmail.com> Would you happen to have anything to do with the creation of the regex module? Just curious. ;) On October 6, 2015 9:24:40 PM CDT, MRAB wrote: >On 2015-10-07 02:30, Steven D'Aprano wrote: >> On Tue, Oct 06, 2015 at 06:34:16PM +0200, M.-A. Lemburg wrote: >>> On 06.10.2015 21:25, Emil Rosendahl Petersen wrote: >>> > I think string.replace should be changed accept a list as a first >>> > argument. >>> > >>> > That way, if I had this string: >>> > >>> > "There are a lot of undesirable people in this filthy world" >>> > >>> > Then I could do this, replace(['undesirable', 'filthy'], ''), in >case >>> > that's what I wanted to do. >>> > >>> > Now, string.replace doesn't accept a list as its first argument, >and >>> > complains about implicit conversion. >> >> [Emil] >>> > Is there any great obstacle to just having the function loop over >that >>> > list, calling itself in case we get a list argument instead of a >str? >> >> Looping over each replacement item is the wrong solution. Think of >> the result when one of the search strings is a substring of the >> replacement: >> >> py> source = "I ate a chicken salad, and she had a ham sandwich." >> py> for term in ["ham", "turkey", "chicken", "spam"]: >> ... source = source.replace(term, "spam and cheese") >> ... >> py> print(source) >> I ate a spam and cheese and cheese salad, and she had a spam and >cheese >> and cheese sandwich. >> >> >> You need to be a bit more careful about how to do the replacements. >> >> >>> > Doesn't that seem like the more obvious behaviour? To me the >results of >>> > running the above code should be unsurprising, if this change was >>> > implemented: "there are a lot of people in this world". >> >> [MAL] >>> I think the "one obvious way" of doing a multi-replace is to >>> use the re module, since implementing this efficiently is >>> non-trivial. >>> >>> String methods are meant to be basic (high performance) >>> operations. >> >> A similar issue was discussed last month, in the context of str.split >> rather than replace, and I talked about the pitfalls of using the re >> module: >> >> >https://mail.python.org/pipermail/python-ideas/2015-September/036586.html >> >> >> The implementation isn't hard, but it's just tricky enough that some >> people will get it wrong, and just useful enough that a helper >function >> will be a good idea. The question is, should that helper function be >a >> string method, in the standard library, or merely something that you >add >> to your own projects? >> >> def replace_all(source, old, new, count=None): >> if isinstance(old, str): >> return source.replace(old, new, count) >> elif isinstance(old, tuple) >> regex = '|'.join(re.escape(s) for s in old) >> return new.join(re.split(regex, source, count)) >> >> >Again, using the regex module, you can split on a named list, without >having to worry about sorting or escaping the items. :-) > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. CURRENTLY LISTENING TO: Prison of Beauty (Kirby Triple Deluxe) by Jun Ishikawa, Hirokazu Ando From guido at python.org Wed Oct 7 06:05:12 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Oct 2015 21:05:12 -0700 Subject: [Python-ideas] PEP 506 (secrets module) and token functions In-Reply-To: <20151007020745.GE28222@ando.pearwood.info> References: <20150919181612.GT31152@ando.pearwood.info> <20150926130715.GG23642@ando.pearwood.info> <20151006160024.GA28222@ando.pearwood.info> <20151007020745.GE28222@ando.pearwood.info> Message-ID: On Tue, Oct 6, 2015 at 7:07 PM, Steven D'Aprano wrote: > On Tue, Oct 06, 2015 at 11:26:11AM -0700, Guido van Rossum wrote: > > This is already looking good. An additional advantage to having a new > > module (as opposed to changing random) is that it could easily be > > backported as a PyPI package, all the way back to Python 2.7. > > > > I do still think that having a concrete proposal for what should > > (initially) go into secrets.py would make for a more compelling PEP. > > > Thanks Guido. > > I'm not sure how much more concrete a proposal you are looking for. The > PEP now lists a sample implementation. I've described it as > "pseudo-code" only to indicate that it may be incomplete (e.g. missing > some imports to make it work, lacking in error checking). It also lacks > docstrings and tests, but otherwise I think it is most of the module. > Hm... I totally did not find that when I read the PEP and even just now I almost missed it! What I had expected was something that could be directly committed into the stlib, similar to the statistics.py module you contributed. But apparently it's so small it could be inlined and overlooked! :-) Maybe you could clean it up, write some tests, and publish it somewhere? (Not sure if you do GitHub. :-) One bikeshed: maybe we should keep only randrange() and drop the confusing randint()? > It's actually not very large, because most of the implementation is > elsewhere (e.g. the random module). > > Is there something else the PEP should include? Any other requested > functions? There have been a few functions suggested that were > requested, e.g. password generation. > > https://www.python.org/dev/peps/pep-0506/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed Oct 7 08:34:58 2015 From: random832 at fastmail.com (Random832) Date: Wed, 07 Oct 2015 02:34:58 -0400 Subject: [Python-ideas] string.replace should accept a list as a first argument In-Reply-To: <3B4C7819-0C14-4737-8D43-DFFD6A7A666E@gmail.com> References: <56140932.1090700@mail.de> <3B4C7819-0C14-4737-8D43-DFFD6A7A666E@gmail.com> Message-ID: <1444199698.676568.403463313.347A9A6E@webmail.messagingengine.com> On Tue, Oct 6, 2015, at 22:19, Ryan Gonzalez wrote: > Isn't that kind of like PyParsing with a DFA? > > ...and that's a really cool idea. Completely coincidentally, I've been > working on a JIT-ted regex library in C. There's no parser yet, and I was > planning on de-exposing the internal structures, but, after reading this, > I'll probably just leave it exposed. Be careful about validating the resulting structure, though. Circular references could be dangerous. I did manage to find the lisp stuff I mentioned: http://scsh.net/docu/html/man-Z-H-7.html http://www.ccs.neu.edu/home/shivers/papers/sre.txt http://srfi.schemers.org/srfi-115/srfi-115.html https://common-lisp.net/~loliveira/ediware/cl-ppcre/doc/ I think the scsh one was the one I'd actually seen before posting the idea. From paddy3118 at gmail.com Wed Oct 7 08:31:42 2015 From: paddy3118 at gmail.com (Paddy3118) Date: Tue, 6 Oct 2015 23:31:42 -0700 (PDT) Subject: [Python-ideas] Rosetta Code on Python.org site? Message-ID: Just saw what the Perl 6 site does to punt the curious to their entries on the Rosetta Code site here: http://perl6.org/community/rosettacode and I thought maybe we could do something similar on Python.org? I know that the core development team of Perl 6 have put a lot of effort into their RC entries, and over the years I have seen evidence of tasks being used to highlight areas for subsequent improvement in Perl 6. I don't *recognise *such a contribution from core development Pythoneers on RC but members of the Python community have made great efforts in making Python solutions available for most tasks on RC in a mixture of Python 2 and 3 and also showing greater use of our interactive command line interface/REPL for solutions which other languages may have, but rarely show unless the task asks specifically for an answer using a REPL. *In short:* Python on RC has good code examples - why not advertise that fact on Python.org in a similar way to the Perl 6 site? *Disclosure: *I should add that I am an administrator, task writer and one of the Python code contributors to the Rosetta Code site. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 7 13:35:05 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Oct 2015 21:35:05 +1000 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: On 4 October 2015 at 08:25, Andrew Barnert wrote: > Nick's suggestion of having it do %-formatting makes sense. Yes, it means > that {count:03} is an error and you need '{count:03d}', which is > inconsistent with f-strings. But that seems like a much less serious problem > than bytes formatting not being able to handle bytes. Exactly, if someone is mistakenly thinking bf"{header}{content}{footer}" is equivalent to f"{header}{content}{footer}".encode(), they're likely to get immediate noisy errors when they start trying to format fields. The parallel I'd attempt to draw is that: f"{header}{content}{footer}" is to "{}{}{}".format(header, content, footer) as: bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % (header, content, footer) To make the behaviour clearer in the latter case, it may be reasonable to *require* an explicit field format code, since that corresponds more closely to the mandatory field format codes in mod-formatting. I'm not sold on the idea of a struct.pack conversion specifier - if we added binary format strings, I think it would be better to start with explicit "pack(value, format)" expressions, and see how that goes for a release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Wed Oct 7 14:34:41 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 7 Oct 2015 05:34:41 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: On Oct 7, 2015, at 04:35, Nick Coghlan wrote: > >> On 4 October 2015 at 08:25, Andrew Barnert wrote: >> Nick's suggestion of having it do %-formatting makes sense. Yes, it means >> that {count:03} is an error and you need '{count:03d}', which is >> inconsistent with f-strings. But that seems like a much less serious problem >> than bytes formatting not being able to handle bytes. > > Exactly, if someone is mistakenly thinking > bf"{header}{content}{footer}" is equivalent to > f"{header}{content}{footer}".encode(), they're likely to get immediate > noisy errors when they start trying to format fields. Except that multiple people in this thread are saying that'd exactly what it should mean (which I think is a very bad idea). > The parallel I'd attempt to draw is that: > > f"{header}{content}{footer}" is to "{}{}{}".format(header, content, footer) > > as: > > bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % > (header, content, footer) > > To make the behaviour clearer in the latter case, it may be reasonable > to *require* an explicit field format code, since that corresponds > more closely to the mandatory field format codes in mod-formatting. Are you suggestive that if a format specifier is given, it must include the format code (which seems perfectly reasonable to me--guessing that :3 means %3b is likely to be wrong more often than it's right?), or that a format specifier must always be given, with no default to :b (which seems more obtrusive and solves less of a problem). From guido at python.org Wed Oct 7 18:25:44 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Oct 2015 09:25:44 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: I think bf'...' should be compared to b'...' % rather than to f'...'. IOW bf'...' is to f'...' as b'...'% is to '...'%. On Wed, Oct 7, 2015 at 5:34 AM, Andrew Barnert wrote: > On Oct 7, 2015, at 04:35, Nick Coghlan wrote: > > > >> On 4 October 2015 at 08:25, Andrew Barnert wrote: > >> Nick's suggestion of having it do %-formatting makes sense. Yes, it > means > >> that {count:03} is an error and you need '{count:03d}', which is > >> inconsistent with f-strings. But that seems like a much less serious > problem > >> than bytes formatting not being able to handle bytes. > > > > Exactly, if someone is mistakenly thinking > > bf"{header}{content}{footer}" is equivalent to > > f"{header}{content}{footer}".encode(), they're likely to get immediate > > noisy errors when they start trying to format fields. > > Except that multiple people in this thread are saying that'd exactly what > it should mean (which I think is a very bad idea). > > > The parallel I'd attempt to draw is that: > > > > f"{header}{content}{footer}" is to "{}{}{}".format(header, content, > footer) > > > > as: > > > > bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % > > (header, content, footer) > > > > To make the behaviour clearer in the latter case, it may be reasonable > > to *require* an explicit field format code, since that corresponds > > more closely to the mandatory field format codes in mod-formatting. > > Are you suggestive that if a format specifier is given, it must include > the format code (which seems perfectly reasonable to me--guessing that :3 > means %3b is likely to be wrong more often than it's right?), or that a > format specifier must always be given, with no default to :b (which seems > more obtrusive and solves less of a problem). > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Oct 7 19:53:08 2015 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 07 Oct 2015 13:53:08 -0400 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: <56155C04.3000908@trueblade.com> On 10/07/2015 12:25 PM, Guido van Rossum wrote: > I think bf'...' should be compared to b'...' % rather than to f'...'. > IOW bf'...' is to f'...' as b'...'% is to '...'%. I'm leaning this way, at least in the sense of "there's a fixed number of known types supported, and there's no extensible protocol involved. Eric. > > On Wed, Oct 7, 2015 at 5:34 AM, Andrew Barnert > wrote: > > On Oct 7, 2015, at 04:35, Nick Coghlan > wrote: > > > >> On 4 October 2015 at 08:25, Andrew Barnert > wrote: > >> Nick's suggestion of having it do %-formatting makes sense. Yes, it means > >> that {count:03} is an error and you need '{count:03d}', which is > >> inconsistent with f-strings. But that seems like a much less serious problem > >> than bytes formatting not being able to handle bytes. > > > > Exactly, if someone is mistakenly thinking > > bf"{header}{content}{footer}" is equivalent to > > f"{header}{content}{footer}".encode(), they're likely to get immediate > > noisy errors when they start trying to format fields. > > Except that multiple people in this thread are saying that'd exactly > what it should mean (which I think is a very bad idea). > > > The parallel I'd attempt to draw is that: > > > > f"{header}{content}{footer}" is to "{}{}{}".format(header, content, footer) > > > > as: > > > > bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % > > (header, content, footer) > > > > To make the behaviour clearer in the latter case, it may be reasonable > > to *require* an explicit field format code, since that corresponds > > more closely to the mandatory field format codes in mod-formatting. > > Are you suggestive that if a format specifier is given, it must > include the format code (which seems perfectly reasonable to > me--guessing that :3 means %3b is likely to be wrong more often than > it's right?), or that a format specifier must always be given, with > no default to :b (which seems more obtrusive and solves less of a > problem). > > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From guido at python.org Wed Oct 7 19:58:37 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Oct 2015 10:58:37 -0700 Subject: [Python-ideas] Binary f-strings In-Reply-To: <56155C04.3000908@trueblade.com> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <56155C04.3000908@trueblade.com> Message-ID: Of course that would still leave the door open for struct.pack support (maybe recognized by having the string start with <,=, > or @). Pro: everybody who currently uses struct.pack will love it. Con: the struct.pack mini-language is pretty inscrutable if you don't already know it. (And no, I don't propose to invent a different mini-language -- it's just easier to figure out where to find docs for this when the code explicitly imports the struct module.) On Wed, Oct 7, 2015 at 10:53 AM, Eric V. Smith wrote: > On 10/07/2015 12:25 PM, Guido van Rossum wrote: > > I think bf'...' should be compared to b'...' % rather than to f'...'. > > IOW bf'...' is to f'...' as b'...'% is to '...'%. > > I'm leaning this way, at least in the sense of "there's a fixed number > of known types supported, and there's no extensible protocol involved. > > Eric. > > > > > On Wed, Oct 7, 2015 at 5:34 AM, Andrew Barnert > > wrote: > > > > On Oct 7, 2015, at 04:35, Nick Coghlan > > wrote: > > > > > >> On 4 October 2015 at 08:25, Andrew Barnert > wrote: > > >> Nick's suggestion of having it do %-formatting makes sense. Yes, > it means > > >> that {count:03} is an error and you need '{count:03d}', which is > > >> inconsistent with f-strings. But that seems like a much less > serious problem > > >> than bytes formatting not being able to handle bytes. > > > > > > Exactly, if someone is mistakenly thinking > > > bf"{header}{content}{footer}" is equivalent to > > > f"{header}{content}{footer}".encode(), they're likely to get > immediate > > > noisy errors when they start trying to format fields. > > > > Except that multiple people in this thread are saying that'd exactly > > what it should mean (which I think is a very bad idea). > > > > > The parallel I'd attempt to draw is that: > > > > > > f"{header}{content}{footer}" is to "{}{}{}".format(header, > content, footer) > > > > > > as: > > > > > > bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % > > > (header, content, footer) > > > > > > To make the behaviour clearer in the latter case, it may be > reasonable > > > to *require* an explicit field format code, since that corresponds > > > more closely to the mandatory field format codes in mod-formatting. > > > > Are you suggestive that if a format specifier is given, it must > > include the format code (which seems perfectly reasonable to > > me--guessing that :3 means %3b is likely to be wrong more often than > > it's right?), or that a format specifier must always be given, with > > no default to :b (which seems more obtrusive and solves less of a > > problem). > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido ) > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Oct 7 20:01:36 2015 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 07 Oct 2015 14:01:36 -0400 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <56155C04.3000908@trueblade.com> Message-ID: <56155E00.9090804@trueblade.com> On 10/07/2015 01:58 PM, Guido van Rossum wrote: > Of course that would still leave the door open for struct.pack support > (maybe recognized by having the string start with <,=, > or @). Pro: > everybody who currently uses struct.pack will love it. Con: the > struct.pack mini-language is pretty inscrutable if you don't already > know it. (And no, I don't propose to invent a different mini-language -- > it's just easier to figure out where to find docs for this when the code > explicitly imports the struct module.) Right. I think Steve Dower's idea of :p switching to struct.pack mode is reasonable. But as Nick says, we don't need to add it on day 1. Eric. > > On Wed, Oct 7, 2015 at 10:53 AM, Eric V. Smith > wrote: > > On 10/07/2015 12:25 PM, Guido van Rossum wrote: > > I think bf'...' should be compared to b'...' % rather than to f'...'. > > IOW bf'...' is to f'...' as b'...'% is to '...'%. > > I'm leaning this way, at least in the sense of "there's a fixed number > of known types supported, and there's no extensible protocol involved. > > Eric. > > > > > On Wed, Oct 7, 2015 at 5:34 AM, Andrew Barnert > > >> wrote: > > > > On Oct 7, 2015, at 04:35, Nick Coghlan > > >> wrote: > > > > > >> On 4 October 2015 at 08:25, Andrew Barnert > > >> wrote: > > >> Nick's suggestion of having it do %-formatting makes sense. > Yes, it means > > >> that {count:03} is an error and you need '{count:03d}', > which is > > >> inconsistent with f-strings. But that seems like a much > less serious problem > > >> than bytes formatting not being able to handle bytes. > > > > > > Exactly, if someone is mistakenly thinking > > > bf"{header}{content}{footer}" is equivalent to > > > f"{header}{content}{footer}".encode(), they're likely to get > immediate > > > noisy errors when they start trying to format fields. > > > > Except that multiple people in this thread are saying that'd > exactly > > what it should mean (which I think is a very bad idea). > > > > > The parallel I'd attempt to draw is that: > > > > > > f"{header}{content}{footer}" is to > "{}{}{}".format(header, content, footer) > > > > > > as: > > > > > > bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % > > > (header, content, footer) > > > > > > To make the behaviour clearer in the latter case, it may be > reasonable > > > to *require* an explicit field format code, since that > corresponds > > > more closely to the mandatory field format codes in > mod-formatting. > > > > Are you suggestive that if a format specifier is given, it must > > include the format code (which seems perfectly reasonable to > > me--guessing that :3 means %3b is likely to be wrong more > often than > > it's right?), or that a format specifier must always be given, > with > > no default to :b (which seems more obtrusive and solves less of a > > problem). > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido > ) > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > -- > --Guido van Rossum (python.org/~guido ) From eric at trueblade.com Wed Oct 7 23:27:11 2015 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 7 Oct 2015 17:27:11 -0400 Subject: [Python-ideas] Binary f-strings In-Reply-To: <56155E00.9090804@trueblade.com> References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> <56155C04.3000908@trueblade.com> <56155E00.9090804@trueblade.com> Message-ID: <5022E7D0-6783-45B7-89BE-C58B166C3FFC@trueblade.com> > On Oct 7, 2015, at 2:01 PM, Eric V. Smith wrote: > >> On 10/07/2015 01:58 PM, Guido van Rossum wrote: >> Of course that would still leave the door open for struct.pack support >> (maybe recognized by having the string start with <,=, > or @). Pro: >> everybody who currently uses struct.pack will love it. Con: the >> struct.pack mini-language is pretty inscrutable if you don't already >> know it. (And no, I don't propose to invent a different mini-language -- >> it's just easier to figure out where to find docs for this when the code >> explicitly imports the struct module.) > > Right. I think Steve Dower's idea of :p switching to struct.pack mode is > reasonable. But as Nick says, we don't need to add it on day 1. Make that "!p". Eric. > > Eric. > >> >> On Wed, Oct 7, 2015 at 10:53 AM, Eric V. Smith > > wrote: >> >>> On 10/07/2015 12:25 PM, Guido van Rossum wrote: >>> I think bf'...' should be compared to b'...' % rather than to f'...'. >>> IOW bf'...' is to f'...' as b'...'% is to '...'%. >> >> I'm leaning this way, at least in the sense of "there's a fixed number >> of known types supported, and there's no extensible protocol involved. >> >> Eric. >> >>> >>> On Wed, Oct 7, 2015 at 5:34 AM, Andrew Barnert >>> >> wrote: >>> >>> On Oct 7, 2015, at 04:35, Nick Coghlan >>> >> wrote: >>>> >>>>> On 4 October 2015 at 08:25, Andrew Barnert >> >> >> wrote: >>>>> Nick's suggestion of having it do %-formatting makes sense. >> Yes, it means >>>>> that {count:03} is an error and you need '{count:03d}', >> which is >>>>> inconsistent with f-strings. But that seems like a much >> less serious problem >>>>> than bytes formatting not being able to handle bytes. >>>> >>>> Exactly, if someone is mistakenly thinking >>>> bf"{header}{content}{footer}" is equivalent to >>>> f"{header}{content}{footer}".encode(), they're likely to get >> immediate >>>> noisy errors when they start trying to format fields. >>> >>> Except that multiple people in this thread are saying that'd >> exactly >>> what it should mean (which I think is a very bad idea). >>> >>>> The parallel I'd attempt to draw is that: >>>> >>>> f"{header}{content}{footer}" is to >> "{}{}{}".format(header, content, footer) >>>> >>>> as: >>>> >>>> bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % >>>> (header, content, footer) >>>> >>>> To make the behaviour clearer in the latter case, it may be >> reasonable >>>> to *require* an explicit field format code, since that >> corresponds >>>> more closely to the mandatory field format codes in >> mod-formatting. >>> >>> Are you suggestive that if a format specifier is given, it must >>> include the format code (which seems perfectly reasonable to >>> me--guessing that :3 means %3b is likely to be wrong more >> often than >>> it's right?), or that a format specifier must always be given, >> with >>> no default to :b (which seems more obtrusive and solves less of a >>> problem). >>> >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido >> ) >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido ) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ncoghlan at gmail.com Thu Oct 8 09:39:19 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Oct 2015 17:39:19 +1000 Subject: [Python-ideas] Binary f-strings In-Reply-To: References: <56089692.1080303@trueblade.com> <560E9292.6040304@mail.de> <560E9429.9040205@trueblade.com> <560EA632.80805@mail.de> Message-ID: On 7 October 2015 at 22:34, Andrew Barnert wrote: > On Oct 7, 2015, at 04:35, Nick Coghlan wrote: >> The parallel I'd attempt to draw is that: >> >> f"{header}{content}{footer}" is to "{}{}{}".format(header, content, footer) >> >> as: >> >> bf"{header:b}{content:b}{footer:b}" would be to b"%b%b%b" % >> (header, content, footer) >> >> To make the behaviour clearer in the latter case, it may be reasonable >> to *require* an explicit field format code, since that corresponds >> more closely to the mandatory field format codes in mod-formatting. > > Are you suggestive that if a format specifier is given, it must include the format code (which seems perfectly reasonable to me--guessing that :3 means %3b is likely to be wrong more often than it's right?), or that a format specifier must always be given, with no default to :b (which seems more obtrusive and solves less of a problem). I was thinking the latter, but your idea of ":b" being implied only if there's no format specifier at all (and otherwise requiring an explicit "b" or other format code) might be better. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From demianbrecht at gmail.com Fri Oct 9 22:21:12 2015 From: demianbrecht at gmail.com (Demian Brecht) Date: Fri, 9 Oct 2015 13:21:12 -0700 Subject: [Python-ideas] Utility to override custom signal handlers Message-ID: Sometimes it's useful to be able to reset a select number of overridden signal handlers temporarily to allow for default handling. While trivial, might there be use for such a utility existing in the signal module? Something to the effect of: @contextmanager def default_sighandlers(*signums): handlers = [] for signum in signums: handlers.append((signum, signal.getsignal(signum))) signal.signal( signum, signal.SIG_DFL if signum != signal.SIGINT else signal.default_int_handler) try: yield finally: for signum, handler in handlers: signal.signal(signum, handler) Thoughts? From andre.roberge at gmail.com Fri Oct 9 22:22:43 2015 From: andre.roberge at gmail.com (Andre Roberge) Date: Fri, 9 Oct 2015 17:22:43 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations Message-ID: Summary: Visual environments based on using simple functions without using variables have been found to be useful to introduce programming concepts to young learners. While Python is an ideal language for beginners, the Python construct for iterations, namely for var in range(n): # block # of # code does require a variable and is needlessly complicated for beginners. A simpler construct like: repeat n: # n is a specific integer # block # of # code would be useful to have in such contexts. Other keywords could be chosen instead of "repeat" used above. ==== For young (and perhaps not so young) students visual environments are an ideal choice to learn programming. A well-known example of a visual environment for beginners is Karel the Robot, invented by Richard Pattis in 1981. In this environment, a robot can perform a limited number of basic functions which can be combined to create complex tasks. In the original Karel the Robot, variables were not allowed. However, new procedures could be created. One essential concept that can be demonstrated in such worlds is that of iterations. In the original Karel, this was done using the following syntax: ITERATE n TIMES single_instruction or ITERATE n TIMES BEGIN single_instruction; other_instruction; END where "n" is a specific integer. In the above, indentation has been introduced only to make the code easier to read. A popular Python-like implementation of Pattis' idea is Guido van Robot [1], also known as GvR. In GvR, iteration is done as follows: do n: # block # of # code indented like Python To do something equivalent in Python requires to use: for var in range(n): # block # of # code which requires to use variables, which are not part of the original Karel the Robot philosophy, as well as an additional builtin function. I would argue that, for beginners, this is needlessly complicated. A better alternative would be to use a similar syntax to that used by Guido van Robot for this case. This would require to either introduce a new keyword (such as "do", "repeat", "iterate", or "loop") repeat n: # block # of # code An advantage of using one of these keywords would be for easier readability (for English speakers). The disadvantage is that some existing Python programs might use variables with the names suggested above. An alternative would be to reuse the "for" keyword by adding this new construct to the grammar: for n: # block # of # code This last choice, while perhaps not as immediately readable to English speakers as some of the previous ones, would have the advantage of not requiring any change to any existing Python programs; it could thus be implemented rather quickly without needing some transition time and the use of the __future__ module. There exists at least 4 different pure Python implementations of Karel the Robot where this new construct could, in principle, be immediately put to use: RUR-PLE [2], Rurple NG [3], both of which are desktop versions, Reeborg's World [4], a web version using Brython as a Python interpreter, and Monty Karel [5], an Eclipse plugin which is targeted at older students as it uses a not so simple OOP syntax even for the simplest programs. Young learners using any of the first three such implementations would likely benefit if they could learn programming first using this simpler construct. Andr? Roberge [1] Guido van Robot: http://gvr.sourceforge.net/documents/languageRef/gvr.html [2] RUR-PLE: https://code.google.com/p/rur-ple/ [3] Rurple NG: http://www.lshift.net/downloads/dev.lshift.net/paul/rurple/manual.html [4] Reeborg's World documentation: http://reeborg.ca/docs/en/index.html World itself: http://reeborg.ca/world.html Python tutorial: http://reeborg.ca/docs/begin_py_en/ [5] Monty Karel: http://www.csis.pace.edu/~bergin/MontyKarel/MontyInEclipse.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri Oct 9 22:27:53 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 09 Oct 2015 15:27:53 -0500 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: Message-ID: <90392E0C-EFB5-4DEB-BB69-24B44FC72052@gmail.com> Just use: for _ in range(n): # magic here and tell them they'll figure out what the _ is later on. That's how I learned it... BTW, that Guido van Robot thing is kind of funny... :D On October 9, 2015 3:22:43 PM CDT, Andre Roberge wrote: >Summary: Visual environments based on using >simple functions without using variables have been found >to be useful to introduce programming concepts to young >learners. While Python is an ideal language for beginners, >the Python construct for iterations, namely > >for var in range(n): > # block > # of > # code > >does require a variable and is needlessly complicated for >beginners. A simpler construct like: > >repeat n: # n is a specific integer > # block > # of > # code > >would be useful to have in such contexts. Other keywords could >be chosen instead of "repeat" used above. > >==== > >For young (and perhaps not so young) students >visual environments are an ideal choice to learn programming. >A well-known example of a visual environment for beginners >is Karel the Robot, invented by Richard Pattis in 1981. >In this environment, a robot can perform a limited number >of basic functions which can be combined to create >complex tasks. > >In the original Karel the Robot, variables were not >allowed. However, new procedures could be created. > >One essential concept that can be demonstrated in such worlds >is that of iterations. In the original Karel, this was >done using the following syntax: > >ITERATE n TIMES > single_instruction > >or > >ITERATE n TIMES > BEGIN > single_instruction; > other_instruction; > END > >where "n" is a specific integer. In the above, indentation >has been introduced only to make the code easier to read. > >A popular Python-like implementation of Pattis' idea >is Guido van Robot [1], also known as GvR. >In GvR, iteration is done as follows: > >do n: > # block > # of > # code indented like Python > >To do something equivalent in Python requires to use: > >for var in range(n): > # block > # of > # code > >which requires to use variables, which are not part of >the original Karel the Robot philosophy, as well as an >additional builtin function. I would argue that, >for beginners, this is needlessly complicated. >A better alternative would be to use a similar >syntax to that used by Guido van Robot for this case. >This would require to either introduce a new keyword >(such as "do", "repeat", "iterate", or "loop") > >repeat n: > # block > # of > # code > >An advantage of using one of these keywords would >be for easier readability (for English speakers). >The disadvantage is that some existing Python programs >might use variables with the names suggested >above. > >An alternative would be to reuse the "for" keyword by >adding this new construct to the grammar: > >for n: > # block > # of > # code > >This last choice, while perhaps not as immediately readable >to English speakers as some of the previous ones, would >have the advantage of not requiring any change to >any existing Python programs; it could thus be implemented >rather quickly without needing some transition >time and the use of the __future__ module. > >There exists at least 4 different pure Python implementations of >Karel the Robot where this new construct could, in principle, >be immediately put to use: > >RUR-PLE [2], Rurple NG [3], both of which are desktop versions, >Reeborg's World [4], a web version using Brython as a Python >interpreter, and Monty Karel [5], an Eclipse plugin which >is targeted at older students as it uses a not so simple >OOP syntax even for the simplest programs. >Young learners using any of the first three such implementations >would likely benefit if they could learn programming first >using this simpler construct. > >Andr? Roberge > >[1] Guido van Robot: >http://gvr.sourceforge.net/documents/languageRef/gvr.html > >[2] RUR-PLE: https://code.google.com/p/rur-ple/ > >[3] Rurple NG: >http://www.lshift.net/downloads/dev.lshift.net/paul/rurple/manual.html > >[4] Reeborg's World documentation: >http://reeborg.ca/docs/en/index.html > World itself: http://reeborg.ca/world.html > Python tutorial: http://reeborg.ca/docs/begin_py_en/ > >[5] Monty Karel: >http://www.csis.pace.edu/~bergin/MontyKarel/MontyInEclipse.html > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From python at lucidity.plus.com Fri Oct 9 23:27:11 2015 From: python at lucidity.plus.com (Erik) Date: Fri, 9 Oct 2015 22:27:11 +0100 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <90392E0C-EFB5-4DEB-BB69-24B44FC72052@gmail.com> References: <90392E0C-EFB5-4DEB-BB69-24B44FC72052@gmail.com> Message-ID: <5618312F.8010403@lucidity.plus.com> On 09/10/15 21:27, Ryan Gonzalez wrote: > Just use: > > for _ in range(n): > # magic here > > and tell them they'll figure out what the _ is later on. That's how I learned it... Or, to make it look that little bit more "natural", spell it: for each in range(n): # magic ... and then one day, move on to explain that "each" is actually something you can USE rather than just being a fixed part of the syntax - at which point it can be spelled better according to what it actually is. An alternative suggestion to remove the need for the variable (rather than a new keyword) would be something like allowing the "var in" part to be optioinal: for : # magic ... but the problem with that is that pretty much the ONLY use would be for this sort of "iterate n times" construct or something where you're relying on the iterator to have side-effects (which is pretty nasty): foo = myiter() for foo: bar(foo.getcurrentvalue()) It seems like a lot of effort just to hide a bit of syntax for a very specific use-case when you can just temporarily make that syntax look a bit nicer for your audience. E. From andre.roberge at gmail.com Fri Oct 9 23:51:23 2015 From: andre.roberge at gmail.com (Andre Roberge) Date: Fri, 9 Oct 2015 18:51:23 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <5618312F.8010403@lucidity.plus.com> References: <90392E0C-EFB5-4DEB-BB69-24B44FC72052@gmail.com> <5618312F.8010403@lucidity.plus.com> Message-ID: On Fri, Oct 9, 2015 at 6:27 PM, Erik wrote: > On 09/10/15 21:27, Ryan Gonzalez wrote: > >> Just use: >> >> for _ in range(n): >> # magic here >> >> and tell them they'll figure out what the _ is later on. That's how I >> learned it... >> > > Or, to make it look that little bit more "natural", spell it: > > for each in range(n): > # magic > > ... and then one day, move on to explain that "each" is actually something > you can USE rather than just being a fixed part of the syntax - at which > point it can be spelled better according to what it actually is. > > An alternative suggestion to remove the need for the variable (rather than > a new keyword) would be something like allowing the "var in" part to be > optioinal: > > for : > # magic > > ... but the problem with that is that pretty much the ONLY use would be > for this sort of "iterate n times" construct or something where you're > relying on the iterator to have side-effects (which is pretty nasty): > > foo = myiter() > for foo: > bar(foo.getcurrentvalue()) > > It seems like a lot of effort just to hide a bit of syntax for a very > specific use-case when you can just temporarily make that syntax look a bit > nicer for your audience. I have been using a special builtin alternative (which can be translated into various human languages, as Reeborg's World is meant to support various languages) as def repeat(f, n): for i in range(n): f() and used as repeat(turn_left, 3) as seen here: http://reeborg.ca/docs/begin_py_en/repeat.html (and for the French version: http://reeborg.ca/docs/begin_py_fr/repeat.html ). In smaller tutorials, I sometimes use the standard "for .." construct instead of this repeat() function, but it always feels clumsy for beginners. However, repeat() has its own complexity in that it uses function arguments, which are inexistant in the original Karel the Robot approach and do not need to be used for simple worlds in Reeborg's World. And, just to put things in perspective: I first wrote RUR-PLE in 2004 and have been working on tutorials for it, discussing with many teachers using it during all these years. So, yes, I know I (and others) can make do without this special construct I am suggesting ... but at the cost of a larger hurdle for the students than would be required if the alternative construct existed. Andr? > > > E. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokoproject at gmail.com Sat Oct 10 00:36:20 2015 From: gokoproject at gmail.com (John Wong) Date: Fri, 9 Oct 2015 18:36:20 -0400 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <90392E0C-EFB5-4DEB-BB69-24B44FC72052@gmail.com> <5618312F.8010403@lucidity.plus.com> Message-ID: On Friday, October 9, 2015, Andre Roberge wrote: > > > And, just to put things in perspective: I first wrote RUR-PLE in 2004 and > have been working on tutorials for it, discussing with many teachers using > it during all these years. So, yes, I know I (and others) can make do > without this special construct I am suggesting ... but at the cost of a > larger hurdle for the students than would be required if the alternative > construct existed. > > But also at the cost of introducting new syntaxatic sugar which is literally a few chars fewer, at the expense of teaching and explain using one or another, "longer standard" version and the shortcut, is not going to make students more productive. I think in general people are smart enough to make decision, recognize tricks. Time should spend on educating students on writing code. This feels like micro optimization. I would rather see for key, value in mydict.items(): And for index, element in enumerate(mylist) to be sugar-coated. This is more annoying to type than the problem being discussed. Everyone's mileage is different but I bet the use of items and enumerate is too frequent and yet there is no briefer version. And I also dont think repeat(..) is helpful for a few chars either. Nice to show how to abstract user from detail implementation (e.g complex repeated task). -- Sent from Jeff Dean's printf() mobile console -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Oct 10 01:33:54 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 9 Oct 2015 16:33:54 -0700 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <5618312F.8010403@lucidity.plus.com> References: <90392E0C-EFB5-4DEB-BB69-24B44FC72052@gmail.com> <5618312F.8010403@lucidity.plus.com> Message-ID: <-5740670168466924456@unknownmsgid> > Or, to make it look that little bit more "natural", spell it: > > for each in range(n): > # magic Indeed -- while python's for loop can be (and often is) used to repeat something a certain number of times, what it really is is a way to apply a block of code to each item in an iterable. Making a slightly shorter/easier way to do something n times will just make it harder to "get" what for loops really are later on. And aside from using it as a teaching tool, does one want to loop without using the index all that often to get special syntax? The Karel-style simplified languages may be a great teaching tool -- but that's not what Python is. -CHB > ... and then one day, move on to explain that "each" is actually something you can USE rather than just being a fixed part of the syntax - at which point it can be spelled better according to what it actually is. > > An alternative suggestion to remove the need for the variable (rather than a new keyword) would be something like allowing the "var in" part to be optioinal: > > for : > # magic > > ... but the problem with that is that pretty much the ONLY use would be for this sort of "iterate n times" construct or something where you're relying on the iterator to have side-effects (which is pretty nasty): > > foo = myiter() > for foo: > bar(foo.getcurrentvalue()) > > It seems like a lot of effort just to hide a bit of syntax for a very specific use-case when you can just temporarily make that syntax look a bit nicer for your audience. > > E. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From tjreedy at udel.edu Sat Oct 10 04:30:18 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 9 Oct 2015 22:30:18 -0400 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: Message-ID: On 10/9/2015 4:22 PM, Andre Roberge wrote: > Summary: Visual environments based on using > simple functions without using variables have been found > to be useful to introduce programming concepts to young > learners. While Python is an ideal language for beginners, > the Python construct for iterations, namely > > for var in range(n): > # block > # of > # code > > does require a variable and is needlessly complicated for > beginners. A simpler construct like: > > repeat n: # n is a specific integer > # block > # of > # code > > would be useful to have in such contexts. Other keywords could > be chosen instead of "repeat" used above. ... > do n: > # block > # of > # code indented like Python > > To do something equivalent in Python requires to use: > > for var in range(n): > # block > # of > # code ... > An alternative would be to reuse the "for" keyword by > adding this new construct to the grammar: > > for n: > # block > # of > # code It was once proposed that int n should be an abbreviation for range(n), at least in the context of a for statement, partly on the basis that counts can be defined in set theory as the set of previous counts, or a similar variation, but really for convenience. Guido vetoed the idea. --- I think trying to persuade core developers to mutilate Python a bit, at couple of years in the future, is futile. And unneeded. Instead, translate, say 'repeat n:' into 'for _ in range(n):' with something like def robotrans(codetext): out = [] for i, line in enumerate(codetext.splitlines()): if line.startswith('repeat'): else: out.append(line) return '\n'.join(outlines) You could, for instance, patch IDLE two places (for shell and editor code) to run robotran on user code before compiling. You could also add 'repeat' to the list of keywords for colorizing. I would be willing to help (without any guarantee of code stability). Guido recently approved on idle-dev that we rethink IDLE for beginners. I have already thought about adding the necessary hooks so that education developers like you could define beginner Python dialects, such as you suggest here. This would open a world of possibilities, such as 'move 3' versus 'move(3)', and foreign language keywords. I just started a thread 'Running beginner Pythons'. -- Terry Jan Reedy From breamoreboy at yahoo.co.uk Sat Oct 10 22:11:55 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 10 Oct 2015 21:11:55 +0100 Subject: [Python-ideas] Proposal how to remove all occurrences of a value from a Python list In-Reply-To: <56106147.8000704@btinternet.com> References: <56106147.8000704@btinternet.com> Message-ID: On 04/10/2015 00:14, Rob Cliffe wrote: > > Other things that might be useful (sorry, I have no use cases in mind): > a.remove(1, count) # maximum number of removals, analogous to > aString.replace(old, new, count) Make the default count=1 to preserve the current behaviour and wouldn't everybody be happy? > a remove-like function that does not raise an error if the item is > not present > a remove-like function that returns the mutated list (like > sorted(), as opposed to list.sort() ) > It's not obvious to me how to design good API(s) to do some/all of this. > Rob Cliffe -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From andre.roberge at gmail.com Sun Oct 11 03:44:13 2015 From: andre.roberge at gmail.com (Andre Roberge) Date: Sat, 10 Oct 2015 22:44:13 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: Message-ID: On Fri, Oct 9, 2015 at 11:30 PM, Terry Reedy wrote: > On 10/9/2015 4:22 PM, Andre Roberge wrote: > >> Summary: Visual environments based on using >> simple functions without using variables have been found >> to be useful to introduce programming concepts to young >> learners. While Python is an ideal language for beginners, >> the Python construct for iterations, namely >> >> for var in range(n): >> # block >> # of >> # code >> >> does require a variable and is needlessly complicated for >> beginners. A simpler construct like: >> >> repeat n: # n is a specific integer >> # block >> # of >> # code >> >> would be useful to have in such contexts. Other keywords could >> be chosen instead of "repeat" used above. >> > > ... > > do n: >> # block >> # of >> # code indented like Python >> >> To do something equivalent in Python requires to use: >> >> for var in range(n): >> # block >> # of >> # code >> > > ... > > An alternative would be to reuse the "for" keyword by >> adding this new construct to the grammar: >> >> for n: >> # block >> # of >> # code >> > > > --- > > I think trying to persuade core developers to mutilate Python a bit, at > couple of years in the future, is futile. It is unfortunate, given the original CP4E goal, but not altogether unexpected -- although, I do find the word "mutilate" to be a bit strong. > And unneeded. I respectfully disagree (however see below). The reason is that I don't believe in students having to unlearn something (like a special repeat syntax available only in a specific environment) when they might have been under the impression that it was standard Python syntax. Furthermore, it might instill doubt in their mind about other construct they have learn (are they standard? ... or were they too something only available in the specific environment?...) > Instead, translate, say 'repeat n:' into 'for _ in range(n):' with > something like > > def robotrans(codetext): > out = [] > for i, line in enumerate(codetext.splitlines()): > if line.startswith('repeat'): > or raise SyntaxError with proper details> > else: > out.append(line) > return '\n'.join(outlines) > > You could, for instance, patch IDLE two places (for shell and editor code) > to run robotran on user code before compiling. You could also add 'repeat' > to the list of keywords for colorizing. I would be willing to help > (without any guarantee of code stability). > I don't use IDLE. To make things as easy as possible for beginners, I have moved away from desktop programs (RUR-PLE) which require a download, to a purely web version, accessible via a single URL. However, prompted by your suggestion, I did implement a simplified syntax for iteration (tentatively using "repeat"). If you click on the link below, it will bring up a concrete example, ready to run. (Note: the syntax used in the example may not work in the future ... say, in a few days from now, after I discuss with other users of my site about this new possible syntax.) If I do end up keeping something like this, I'll have to make sure it is flagged very clearly as being non-standard, so that the students are properly warned. http://reeborg.ca/world_dev.html?proglang=python-en&world=%7B%0A%20%20%22robots%22%3A%20%5B%0A%20%20%20%20%7B%0A%20%20%20%20%20%20%22x%22%3A%201%2C%0A%20%20%20%20%20%20%22y%22%3A%201%2C%0A%20%20%20%20%20%20%22orientation%22%3A%200%2C%0A%20%20%20%20%20%20%22_prev_x%22%3A%201%2C%0A%20%20%20%20%20%20%22_prev_y%22%3A%201%2C%0A%20%20%20%20%20%20%22_prev_orientation%22%3A%200%2C%0A%20%20%20%20%20%20%22objects%22%3A%20%7B%7D%0A%20%20%20%20%7D%0A%20%20%5D%2C%0A%20%20%22walls%22%3A%20%7B%7D%2C%0A%20%20%22description%22%3A%20%22%22%2C%0A%20%20%22small_tiles%22%3A%20false%2C%0A%20%20%22rows%22%3A%2012%2C%0A%20%20%22cols%22%3A%2014%0A%7D&editor=from%20library%20import%20left%2C%20right%0A%0Arepeat%204%3A%0A%20%20%20%20move()%0A%20%20%20%20left()%0A%0Aright()%0A&library=left%20%3D%20turn_left%0A%20%20%20%20%0Adef%20right()%3A%0A%20%20%20%20repeat%203%3A%0A%20%20%20%20%20%20%20%20turn_left()%0A > Guido recently approved on idle-dev that we rethink IDLE for beginners. I > have already thought about adding the necessary hooks so that education > developers like you could define beginner Python dialects, such as you > suggest here. This would open a world of possibilities, such as 'move 3' > versus 'move(3)', and foreign language keywords. I just started a thread > 'Running beginner Pythons'. I'll have a look. However, as I mentioned above, I'm not keen on introducing what amounts to a potentially completely different language. Python is a fantastic language for beginners (and advanced) programmers ... I was just hoping to make it a tiny bit easier to learn for complete beginners who are introduced to Python in visual environments (like the turtle module or a Karel the Robot clone). Andr? Roberge > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.team.python at gmail.com Sun Oct 11 04:55:58 2015 From: ian.team.python at gmail.com (Ian) Date: Sun, 11 Oct 2015 13:55:58 +1100 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <-5740670168466924456@unknownmsgid> References: <90392E0C-EFB5-4DEB-BB69-24B44FC72052@gmail.com> <5618312F.8010403@lucidity.plus.com> <-5740670168466924456@unknownmsgid> Message-ID: <5619CFBE.3050700@gmail.com> if anything is changed at all, then the for : # magic seems to have the least impact and best addresses the original goal. repeat n: # saves 4 characters vs for range(n): # keeps syntax consistent Which (consistency vs save 4 characters) would you favour for a beginner? Not to mention that introducing a new keyword introduces many challenges. >> Or, to make it look that little bit more "natural", spell it: >> >> for each in range(n): >> # magic > Indeed -- while python's for loop can be (and often is) used to repeat > something a certain number of times, what it really is is a way to > apply a block of code to each item in an iterable. > > Making a slightly shorter/easier way to do something n times will just > make it harder to "get" what for loops really are later on. > > And aside from using it as a teaching tool, does one want to loop > without using the index all that often to get special syntax? > > The Karel-style simplified languages may be a great teaching tool -- > but that's not what Python is. > > -CHB > > > >> ... and then one day, move on to explain that "each" is actually something you can USE rather than just being a fixed part of the syntax - at which point it can be spelled better according to what it actually is. >> >> An alternative suggestion to remove the need for the variable (rather than a new keyword) would be something like allowing the "var in" part to be optioinal: >> >> for : >> # magic >> >> ... but the problem with that is that pretty much the ONLY use would be for this sort of "iterate n times" construct or something where you're relying on the iterator to have side-effects (which is pretty nasty): >> >> foo = myiter() >> for foo: >> bar(foo.getcurrentvalue()) >> >> It seems like a lot of effort just to hide a bit of syntax for a very specific use-case when you can just temporarily make that syntax look a bit nicer for your audience. >> >> E. >> _______________________________________________ >> From tjreedy at udel.edu Sun Oct 11 06:37:06 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 11 Oct 2015 00:37:06 -0400 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: Message-ID: On 10/10/2015 9:44 PM, Andre Roberge wrote > On Fri, Oct 9, 2015 at 11:30 PM, Terry Reedy > > wrote: > The reason is that I don't believe in students having to unlearn > something (like a special repeat syntax available only in a specific > environment) when they might have been under the impression that it was > standard Python syntax. To avoid mis-impressions, non-standard syntax might get a special syntax coloring, such as an orange 'warning color' background. Or it could be translated line-by-line on entry: enter 'for n' and the line is immediately replaced by 'for _ in range(n):'. (I might like that for my own use ;-). Or if the beginner language is really not Python, don't call it Python. I know I don't know how to teach beginning programming, especially to children, and I doubt anyone really knows the *best* way. So I am interested in facilitating experiments. > Furthermore, it might instill doubt in their > mind about other construct they have learn (are they standard? ... or > were they too something only available in the specific environment?...) I think I understood at a pretty early age that training wheels were a special part of a beginner bike for little kids, only present for learning. > Python is a fantastic language for beginners (and advanced) > programmers ... I was just hoping to make it a tiny bit easier to learn > for complete beginners who are introduced to Python in visual > environments (like the turtle module or a Karel the Robot clone). Your are asking for a special case of a special case of something already present. In my view, it is like a training wheel - very useful when needed, but not so thereafter. Since it is only needed for a short time for young beginners, it only need be present, in my view, in the environments used by such beginners. -- Terry Jan Reedy From stephen at xemacs.org Sun Oct 11 09:48:12 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 11 Oct 2015 16:48:12 +0900 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: Message-ID: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Andre Roberge writes: > >> While Python is an ideal language for beginners, the Python > >> construct for iterations, namely > >> > >> for var in range(n): > >> # suite > >> > >> does require a variable and is needlessly complicated for > >> beginners. A simpler construct like: > >> > >> repeat n: # n is a specific integer > >> # suite > >> > >> would be useful to have in such contexts. Other keywords could > >> be chosen instead of "repeat" used above. I have no objection to such a language, but it ain't Python. It's an important principle of Python that "There's one- and preferably only one -obvious way to do it." While the parenthetical "only one" is violated on occasion, it's still considered important. > >> for var in range(n): > >> # suite I just don't have a problem with this. It seems to me that on the occasions when I've taught programming[2], I've always had the most success when the student has a concrete problem in mind (sometimes they need to write programs to complete another project, but also when they take programming as a required course). Take the traditional make a square in turtle graphics: right = -90 length = 10 def make_square (turtle): for side in ('top', 'right', 'bottom', 'left'): turtle.forward(length) turtle.turn(right) Do you really have students writing code like for _ in range(20): print('yes') where the repetition is exact? > It is unfortunate, given the original CP4E goal, but not altogether > unexpected -- although, I do find the word "mutilate" to be a bit > strong. The goal called "CP4E" as I understood it was to give the *full* power of "computer programming" to "everyone", not to just provide a taste. The *power* of computer programming comes from small but expressive vocabularies, it seems to me. > The reason is that I don't believe in students having to unlearn > something (like a special repeat syntax available only in a > specific environment) when they might have been under the > impression that it was standard Python syntax. So call the new language "Childrens-Python"[1] if you like. Or "Woma" (which is a bit bigger than Children's Python) if that's too close to "Python". > Furthermore, it might instill doubt in their mind about other > construct they have learn (are they standard? ... or were they too > something only available in the specific environment?...) So what? If they're going to do computer programming after they leave your class, they're surely going to encounter "specific environments". So many times I've typed an example into the interpreter only to discover that the author didn't explicitly import a function or identifier from a module, but it looks like it could be a builtin I hadn't used before. > I'll have a look. However, as I mentioned above, I'm not keen on > introducing what amounts to a potentially completely different > language. It doesn't look to me like that's what Terry is talking about, though. I suppose that as usual there will be a "consenting adults" convention that "you could do that with this hook, but please, *not* in front of the children!" Rather, for cases where you don't need the full power of Python's expression syntax, you could eliminate parentheses from function calls and that kind of thing. > Python is a fantastic language for beginners (and advanced) > programmers ... I was just hoping to make it a tiny bit easier to > learn for complete beginners who are introduced to Python in visual > environments (like the turtle module or a Karel the Robot clone). At the cost of requiring every Python programmer to beware of a new and infrequently syntax. Footnotes: [1] http://www.snakeranch.com.au/snake-profiles/childrens-python/ [2] Always at the college level or above, though. From luciano at ramalho.org Sun Oct 11 23:47:10 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Sun, 11 Oct 2015 18:47:10 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: I've felt the same need Andre described, and so far I have not seen a convincing argument against his proposal to accept this syntax: for n: # repeat block n times fd(100) lt(360/n) Where n is any expression that evaluates to an integer. The goal is to repeat some operation N times without creating a spurious variable. I prefer this simpler alternative than Ian's later proposal where n must be an iterable, involving a lot of machinery just to produce a value to be discarded. While we are at it, we could borrow Go's "forever" loop: for: # repeat forever spam() Although I am quite happy with `while True:` for that purpose. Cheers, Luciano On Sun, Oct 11, 2015 at 4:48 AM, Stephen J. Turnbull wrote: > Andre Roberge writes: > > > >> While Python is an ideal language for beginners, the Python > > >> construct for iterations, namely > > >> > > >> for var in range(n): > > >> # suite > > >> > > >> does require a variable and is needlessly complicated for > > >> beginners. A simpler construct like: > > >> > > >> repeat n: # n is a specific integer > > >> # suite > > >> > > >> would be useful to have in such contexts. Other keywords could > > >> be chosen instead of "repeat" used above. > > I have no objection to such a language, but it ain't Python. It's an > important principle of Python that "There's one- and preferably only > one -obvious way to do it." While the parenthetical "only one" is > violated on occasion, it's still considered important. > > > >> for var in range(n): > > >> # suite > > I just don't have a problem with this. > > It seems to me that on the occasions when I've taught programming[2], > I've always had the most success when the student has a concrete > problem in mind (sometimes they need to write programs to complete > another project, but also when they take programming as a required > course). Take the traditional make a square in turtle graphics: > > right = -90 > length = 10 > def make_square (turtle): > for side in ('top', 'right', 'bottom', 'left'): > turtle.forward(length) > turtle.turn(right) > > Do you really have students writing code like > > for _ in range(20): > print('yes') > > where the repetition is exact? > > > It is unfortunate, given the original CP4E goal, but not altogether > > unexpected -- although, I do find the word "mutilate" to be a bit > > strong. > > The goal called "CP4E" as I understood it was to give the *full* power > of "computer programming" to "everyone", not to just provide a taste. > The *power* of computer programming comes from small but expressive > vocabularies, it seems to me. > > > The reason is that I don't believe in students having to unlearn > > something (like a special repeat syntax available only in a > > specific environment) when they might have been under the > > impression that it was standard Python syntax. > > So call the new language "Childrens-Python"[1] if you like. Or > "Woma" (which is a bit bigger than Children's Python) if that's too > close to "Python". > > > Furthermore, it might instill doubt in their mind about other > > construct they have learn (are they standard? ... or were they too > > something only available in the specific environment?...) > > So what? If they're going to do computer programming after they leave > your class, they're surely going to encounter "specific environments". > So many times I've typed an example into the interpreter only to > discover that the author didn't explicitly import a function or > identifier from a module, but it looks like it could be a builtin I > hadn't used before. > > > I'll have a look. However, as I mentioned above, I'm not keen on > > introducing what amounts to a potentially completely different > > language. > > It doesn't look to me like that's what Terry is talking about, though. > I suppose that as usual there will be a "consenting adults" convention > that "you could do that with this hook, but please, *not* in front of > the children!" Rather, for cases where you don't need the full power > of Python's expression syntax, you could eliminate parentheses from > function calls and that kind of thing. > > > Python is a fantastic language for beginners (and advanced) > > programmers ... I was just hoping to make it a tiny bit easier to > > learn for complete beginners who are introduced to Python in visual > > environments (like the turtle module or a Karel the Robot clone). > > At the cost of requiring every Python programmer to beware of a new > and infrequently syntax. > > > Footnotes: > [1] http://www.snakeranch.com.au/snake-profiles/childrens-python/ > > [2] Always at the college level or above, though. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From ben+python at benfinney.id.au Mon Oct 12 00:03:27 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 12 Oct 2015 09:03:27 +1100 Subject: [Python-ideas] Simpler syntax for basic iterations References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: <858u79ukpc.fsf@benfinney.id.au> Luciano Ramalho writes: > I've felt the same need Andre described, and so far I have not seen a > convincing argument against his proposal to accept this syntax: > > for n: # repeat block n times > fd(100) > lt(360/n) > > Where n is any expression that evaluates to an integer. Huh? If ?n? is an expression that evaluates to one integer, then why are you also dividing by that one integer ?n? every time through? I don't claim it will convince you, but I find compelling the fact that this:: for n in range(FOO): fd(100) lt(360/n) already works fine and is quite clear. -- \ ?Science is a way of trying not to fool yourself. The first | `\ principle is that you must not fool yourself, and you are the | _o__) easiest person to fool.? ?Richard P. Feynman, 1964 | Ben Finney From ben+python at benfinney.id.au Mon Oct 12 00:08:13 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 12 Oct 2015 09:08:13 +1100 Subject: [Python-ideas] Simpler syntax for basic iterations References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <858u79ukpc.fsf@benfinney.id.au> Message-ID: <854mhxukhe.fsf@benfinney.id.au> Ben Finney writes: > I don't claim it will convince you, but I find compelling the fact that > this:: > > for n in range(FOO): > fd(100) > lt(360/n) > > already works fine and is quite clear. And, of course, in my haste I divided by zero :-) -- \ ?We cannot solve our problems with the same thinking we used | `\ when we created them.? ?Albert Einstein | _o__) | Ben Finney From luciano at ramalho.org Mon Oct 12 00:13:27 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Sun, 11 Oct 2015 19:13:27 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <858u79ukpc.fsf@benfinney.id.au> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <858u79ukpc.fsf@benfinney.id.au> Message-ID: On Sun, Oct 11, 2015 at 7:03 PM, Ben Finney wrote: > Luciano Ramalho > writes: >> for n: # repeat block n times >> fd(100) >> lt(360/n) >> >> Where n is any expression that evaluates to an integer. > > Huh? If ?n? is an expression that evaluates to one integer, then why are > you also dividing by that one integer ?n? every time through? That's the classic formula for drawing an N-sided polygon in Logo. N can be any positive integer. Maybe this is clearer: for 3: # draw a triangle fd(100) lt(120) > I don't claim it will convince you, but I find compelling the fact that > this:: > > for n in range(FOO): > fd(100) > lt(360/n) > > already works fine and is quite clear. It is quite clear for us. When teaching the first steps of programming to kids with the Turtle module, `for x in range(n):` leaves us in the unfortunate position of having to say: "we won't use that x for anything, and I can't explain what range this until much later..." Cheers, Luciano > > -- > \ ?Science is a way of trying not to fool yourself. The first | > `\ principle is that you must not fool yourself, and you are the | > _o__) easiest person to fool.? ?Richard P. Feynman, 1964 | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From ben+python at benfinney.id.au Mon Oct 12 00:28:22 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 12 Oct 2015 09:28:22 +1100 Subject: [Python-ideas] Simpler syntax for basic iterations References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <858u79ukpc.fsf@benfinney.id.au> Message-ID: <85zizpt4zd.fsf@benfinney.id.au> Luciano Ramalho writes: > That's the classic formula for drawing an N-sided polygon in Logo. N > can be any positive integer. Maybe this is clearer: > > for 3: # draw a triangle > fd(100) > lt(120) Thank you, yes it's much clearer because you're not doing exactly the same division multiple times. So, to follow your clarification, this:: for __ in range(3): forward(100) turn_left(120) already works and is quite clear (though, again, I don't claim it will convince you). > It is quite clear for us. When teaching the first steps of programming > to kids with the Turtle module, `for x in range(n):` leaves us in the > unfortunate position of having to say: "we won't use that x for > anything, and I can't explain what range this until much later..." Yes, the fact you weren't using it confused me too :-) Which is why we advocate a name, ?__?, that stands out and is used only for dummy assignment. -- \ ?How wonderful that we have met with a paradox. Now we have | `\ some hope of making progress.? ?Niels Bohr | _o__) | Ben Finney From guido at python.org Mon Oct 12 00:59:00 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2015 15:59:00 -0700 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <85zizpt4zd.fsf@benfinney.id.au> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <858u79ukpc.fsf@benfinney.id.au> <85zizpt4zd.fsf@benfinney.id.au> Message-ID: Argh! Here's why I don't like adding special syntax to do something N times. I didn't leave this out by accident 25 years ago. Python takes a stance against a common approach to beginners' programming education focused on counting and indexing arrays of data, rooted in machine language and FORTRAN. Standard problems and examples like "print 'hello world' 10 times" or "draw an N-sided polygon" are designed to teach students those counting and indexing skills. It's backwards to request that the language support counting better so those examples will look simpler. Python has so much richer data structures. Create problems to teach those! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From luciano at ramalho.org Mon Oct 12 01:22:28 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Sun, 11 Oct 2015 20:22:28 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <858u79ukpc.fsf@benfinney.id.au> <85zizpt4zd.fsf@benfinney.id.au> Message-ID: On Sun, Oct 11, 2015 at 7:59 PM, Guido van Rossum wrote: > Standard problems and examples like "print 'hello world' 10 times" or "draw > an N-sided polygon" are designed to teach students those counting and > indexing skills. It's backwards to request that the language support > counting better so those examples will look simpler. I agree that counting and indexing dominate many stupid introductory programming exercises. And I praise the design of Python's `for` command every time I teach the language or speak about it. However, in Logo, using a command like the one below, there is absolutely no indexing or counting in sight. REPEAT 3 [FD 100 RT 120] That's why we were advocating for a similar construct in Python. But obviously you're not going to entertain the idea, so I'll drop it. Cheers, Luciano > > Python has so much richer data structures. Create problems to teach those! > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From emile at fenx.com Mon Oct 12 03:59:13 2015 From: emile at fenx.com (Emile van Sebille) Date: Sun, 11 Oct 2015 18:59:13 -0700 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: On 10/11/2015 2:47 PM, Luciano Ramalho wrote: > I've felt the same need Andre described, and so far I have not seen a > convincing argument against his proposal to accept this syntax: > > for n: # repeat block n times > fd(100) > lt(360/n) > > Where n is any expression that evaluates to an integer. Umm... https://www.python.org/dev/peps/pep-0284/ Emile From guido at python.org Mon Oct 12 04:12:03 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2015 19:12:03 -0700 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sun, Oct 11, 2015 at 6:59 PM, Emile van Sebille wrote: > On 10/11/2015 2:47 PM, Luciano Ramalho wrote: > >> I've felt the same need Andre described, and so far I have not seen a >> convincing argument against his proposal to accept this syntax: >> >> for n: # repeat block n times >> fd(100) >> lt(360/n) >> >> Where n is any expression that evaluates to an integer. >> > > Umm... https://www.python.org/dev/peps/pep-0284/ > Heh. "The whole point (15 years ago) of range() was to *avoid* needing syntax to specify a loop over numbers." I guess I'm nothing if not consistent. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From luciano at ramalho.org Mon Oct 12 05:03:48 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Mon, 12 Oct 2015 00:03:48 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sun, Oct 11, 2015 at 11:12 PM, Guido van Rossum wrote: > Heh. "The whole point (15 years ago) of range() was to *avoid* needing > syntax to specify a loop over numbers." I guess I'm nothing if not > consistent. :-) Andre and I are not arguing for a special syntax to loop over numbers. We are arguing for special syntax to repeat a block a number of times. No indexing at all, and no variables need to be involved. for 3: fd(100) lt(120) Best, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From guido at python.org Mon Oct 12 06:43:52 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2015 21:43:52 -0700 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: Well, that's even more limited. On Sunday, October 11, 2015, Luciano Ramalho wrote: > On Sun, Oct 11, 2015 at 11:12 PM, Guido van Rossum > wrote: > > Heh. "The whole point (15 years ago) of range() was to *avoid* needing > > syntax to specify a loop over numbers." I guess I'm nothing if not > > consistent. :-) > > Andre and I are not arguing for a special syntax to loop over numbers. > We are arguing for special syntax to repeat a block a number of times. > No indexing at all, and no variables need to be involved. > > for 3: > fd(100) > lt(120) > > > Best, > > Luciano > > -- > Luciano Ramalho > | Author of Fluent Python (O'Reilly, 2015) > | http://shop.oreilly.com/product/0636920032519.do > | Professor em: http://python.pro.br > | Twitter: @ramalhoorg > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From luciano at ramalho.org Mon Oct 12 06:58:45 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Mon, 12 Oct 2015 01:58:45 -0300 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mon, Oct 12, 2015 at 1:43 AM, Guido van Rossum wrote: > Well, that's even more limited. Yep. That's the whole point. > > > On Sunday, October 11, 2015, Luciano Ramalho wrote: >> >> On Sun, Oct 11, 2015 at 11:12 PM, Guido van Rossum >> wrote: >> > Heh. "The whole point (15 years ago) of range() was to *avoid* needing >> > syntax to specify a loop over numbers." I guess I'm nothing if not >> > consistent. :-) >> >> Andre and I are not arguing for a special syntax to loop over numbers. >> We are arguing for special syntax to repeat a block a number of times. >> No indexing at all, and no variables need to be involved. >> >> for 3: >> fd(100) >> lt(120) >> >> >> Best, >> >> Luciano >> >> -- >> Luciano Ramalho >> | Author of Fluent Python (O'Reilly, 2015) >> | http://shop.oreilly.com/product/0636920032519.do >> | Professor em: http://python.pro.br >> | Twitter: @ramalhoorg > > > > -- > --Guido (mobile) -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From guido at python.org Mon Oct 12 07:05:18 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Oct 2015 22:05:18 -0700 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: It's not very compelling. So let's drop it. On Sunday, October 11, 2015, Luciano Ramalho wrote: > On Mon, Oct 12, 2015 at 1:43 AM, Guido van Rossum > wrote: > > Well, that's even more limited. > > Yep. That's the whole point. > > > > > > > On Sunday, October 11, 2015, Luciano Ramalho > wrote: > >> > >> On Sun, Oct 11, 2015 at 11:12 PM, Guido van Rossum > > >> wrote: > >> > Heh. "The whole point (15 years ago) of range() was to *avoid* needing > >> > syntax to specify a loop over numbers." I guess I'm nothing if not > >> > consistent. :-) > >> > >> Andre and I are not arguing for a special syntax to loop over numbers. > >> We are arguing for special syntax to repeat a block a number of times. > >> No indexing at all, and no variables need to be involved. > >> > >> for 3: > >> fd(100) > >> lt(120) > >> > >> > >> Best, > >> > >> Luciano > >> > >> -- > >> Luciano Ramalho > >> | Author of Fluent Python (O'Reilly, 2015) > >> | http://shop.oreilly.com/product/0636920032519.do > >> | Professor em: http://python.pro.br > >> | Twitter: @ramalhoorg > > > > > > > > -- > > --Guido (mobile) > > > > -- > Luciano Ramalho > | Author of Fluent Python (O'Reilly, 2015) > | http://shop.oreilly.com/product/0636920032519.do > | Professor em: http://python.pro.br > | Twitter: @ramalhoorg > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rustompmody at gmail.com Mon Oct 12 07:37:43 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 12 Oct 2015 11:07:43 +0530 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mon, Oct 12, 2015 at 10:28 AM, Luciano Ramalho wrote: > On Mon, Oct 12, 2015 at 1:43 AM, Guido van Rossum > wrote: > > Well, that's even more limited. > > Yep. That's the whole point. > > In discussions like this there are 3 ? at least ? interlocking parties with their conflicting interests. 1. Professional programmers 2. Language Designers/implementers 3. Programming/CS teachers It is unlikely to find people equal to all these but at least let us be clear that these 3 exist! [For those who find this too long, tl;dr at bottom ] To see that these viewpoints are inherently conflicting consider that a programmer may wish that his language precludes errors ? a natural wish. Translated to the designer camp this means an automatic detection of errors ? also known as the Halting Problem. Which is to show that the programmer's and the language-designer's needs are not just conflicting, they are inherently conflicting And so, in like manner the programming-teachers' needs are in inherent conflict with the needs of professional programmers and language designers. 3-4 decades ago this was widely understood and some sort of compromise was worked out: Beginners studied Pascal (or Basic); professionals lived inside Fortran, Cobol, PL-1 etc. With the coming of C, people began to imagine that C was sufficiently close to Pascal that a one-fit-all solution was possible. The result has been terribly detrimental to CS-edcuation The increasing widespreadness of languages like Python and Haskell may seem to improve this situation. IMHO it only worsens it: Sure Python is better to teach than C++ but its not better enough that its suitable as a teaching-language. [And as Haskell gallops towards Real-World status it also gallops towards C++-dom]. Specifically coming to the suggestion at hand: Pascal got one thing right that all its successors, starting C, got terribly wrong: the need to separate and *do equal justice to* Platonic world Mundane World Values Effects Expressions Statements functions procedures More about this 50 year-old error and a beginning gleam of light: a ACM CS Curriculum 2013 and its prequel Timeline The error of imagining imperative programming is easy to learn comes from people not seeing that imperative languages *as they exist* contain a subset functional language which is ineluctable. Also known as the rhs of assignment statements ? with the '=' being the visible indicator of the infamous von Neumann bottleneck . The functional programming folks seeing this error try to expand that expression world to cover all of programming needs. [Whether the results of these good-intentions eg Haskell are more teachable is another matter!] The alternative (eg Logo and I am guessing Luciano and Andre ) that for the 'atoms' of the statement world, instead of having assignment statements to have turtle-left/right etc. This renders the language useless for general purpose programming but can be a neat intro to algorithmic thinking tl;dr 1. Language designer/implementers may wish to ask themselves whether having a family of easily learnable but slightly inconsistent languages helps or hinders their language. 2. Teachers may wish to re-evaluate their fixed-cost/variable-cost (equivalently batch/interactive cost) outlay in the languages they teach and check whether implementing a cut-down subset of their chosen language doesn't speed up their students and themselves 3. General programmers need to remember ? not only were they noobs at some stage, there is probably something they are learning right now to which noob-ness applies even to them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Oct 12 08:10:24 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Oct 2015 02:10:24 -0400 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: On 10/11/2015 5:47 PM, Luciano Ramalho wrote: > I've felt the same need Andre described, and so far I have not seen a > convincing argument against his proposal to accept this syntax: Your logic is backwards. The default answer to 'add this' is 'No'. The burden of persuasion is on you. The people who have to be persuaded, by their own standards of goodness for Python, are Guido and core developers > > for n: # repeat block n times > fd(100) > lt(360/n) > > Where n is any expression that evaluates to an integer. Andre wanted, I believe, 'for :' to avoid naming numbers, or at least variable quantities. But I am 'not convinced'. The above should be turn = 360/n for n: fd(100) lt(turn) Anyway, very young kids get the concept of 'variable quantity': number of days left to Christmas, number of cookies handed out, etc. > The goal is to repeat some operation N times without creating a > spurious variable. The fraction of for loops using range(n) is small, especially after enumerate was introduced, and especially if n is a constant. The fraction of such loops where the iteration count can never have any use is also small, because if the code has a bug, you will likely want the iteration count. Adding 'print(i, a, k)' to a loop, where a and k change with each loop is, for me, a very common debugging tool. Even without bugs, I believe that displaying iteration counts can help learning and program understanding. Right now, one could step through n = 17 for i in range(n): # i = number of chord already drawn fd(100) lt(360/n) with IDLE's debugger and see the step number increase just before a chord is drawn. Or one might want to add 'label(i)' at the top, where label(i) puts a small circled number near the current position. The more I think about this, the less I am convinced that the fraction of truly, permanently spurious loop counts is enough to support a new syntax. -- Terry Jan Reedy From rustompmody at gmail.com Mon Oct 12 10:01:15 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 12 Oct 2015 01:01:15 -0700 (PDT) Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: <2f77b95c-ffd1-460d-bd72-9991068780ca@googlegroups.com> On Monday, October 12, 2015 at 11:41:06 AM UTC+5:30, Terry Reedy wrote: > > On 10/11/2015 5:47 PM, Luciano Ramalho wrote: > > I've felt the same need Andre described, and so far I have not seen a > > convincing argument against his proposal to accept this syntax: > > Your logic is backwards. The default answer to 'add this' is 'No'. The > burden of persuasion is on you. The people who have to be persuaded, by > their own standards of goodness for Python, are Guido and core developers > Just to clarify my earlier post: I dont regard this suggestion as a serious contender for python. I would however like it that if someone forked and added this to (a probably heavily cut-down) python, that fork be regarded as a friendly-by-default fork In my personal estimate: of such talk 1% will go from being vaporware to something others can actually try out Of that 1%, 0.1% can become a serious competitor for (C)Python > > > > > for n: # repeat block n times > > fd(100) > > lt(360/n) > > > > Where n is any expression that evaluates to an integer. > > Andre wanted, I believe, 'for :' to avoid naming > numbers, or at least variable quantities. But I am 'not convinced'. > The above should be > > turn = 360/n > for n: > fd(100) > lt(turn) > > Anyway, very young kids get the concept of 'variable quantity': number > of days left to Christmas, number of cookies handed out, etc. > > I just attended some lectures in which a gifted teacher used Jupyter (ipython) to give abstruse linear algebra intuitions to kids. So yes a good teacher can convey variables to kids. Is it a good idea to do so? I remember as a 10-11 year old having a hard time going from arithmetic to algebra. Current day imperative languages makes this (in my experience inherent) hardness harder on 2 more counts: - the variable is time-varying - the assignment is denoted with '=' which is a confusing synonym/homonym to the math equality And so the teacher trying to disentangle all this confusingness is to be supported (in my book) Its another matter that python is not a 'for-kids' language so it need not change to incorporate this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Tue Oct 13 00:21:31 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 13 Oct 2015 09:21:31 +1100 Subject: [Python-ideas] Simpler syntax for basic iterations References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> Message-ID: <85mvvnu3ro.fsf@benfinney.id.au> Luciano Ramalho writes: > Andre and I are not arguing for a special syntax to loop over numbers. > We are arguing for special syntax to repeat a block a number of times. That's been clear, at least to my reading of your messages. What you haven't argued is any extraordinary utility of this new syntax, sufficient to overcome the high barrier that is rightly imposed on new syntax. > for 3: > fd(100) > lt(120) Since this is quite clearly expressed in existing Python syntax with:: for __ in range(3): forward(100) turn_left(120) the single use case doesn't IMO argue for benefit sufficient to overcome the enormous burden of new syntax. -- \ ?I'm not a bad guy! I work hard, and I love my kids. So why | `\ should I spend half my Sunday hearing about how I'm going to | _o__) Hell?? ?Homer Simpson | Ben Finney From srkunze at mail.de Tue Oct 13 19:59:55 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 13 Oct 2015 19:59:55 +0200 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <85mvvnu3ro.fsf@benfinney.id.au> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> Message-ID: <561D469B.9020305@mail.de> On 13.10.2015 00:21, Ben Finney wrote: > Since this is quite clearly expressed in existing Python syntax with:: > > for __ in range(3): > forward(100) > turn_left(120) > > the single use case doesn't IMO argue for benefit sufficient to overcome > the enormous burden of new syntax. I still like instead of the cryptic __ or _ for each in range(3): dosomething To me, that reads like a normal English statement. Best, Sven From ben+python at benfinney.id.au Tue Oct 13 21:49:39 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 14 Oct 2015 06:49:39 +1100 Subject: [Python-ideas] Simpler syntax for basic iterations References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> Message-ID: <85eggytup8.fsf@benfinney.id.au> "Sven R. Kunze" writes: > On 13.10.2015 00:21, Ben Finney wrote: > > Since this is quite clearly expressed in existing Python syntax with:: > > > > for __ in range(3): > > forward(100) > > turn_left(120) > > > > the single use case doesn't IMO argue for benefit sufficient to overcome > > the enormous burden of new syntax. > > I still like instead of the cryptic __ or _ > > for each in range(3): > dosomething > > To me, that reads like a normal English statement. What you cite as an advantage is exactly why I'd advise against it. Your example would lead me to ask ?Is this a mistake? ?each? is a normal name, why isn't it being used?? Whereas ?__? is not a normal name, and like other unusual names in Python should lead the reader to say ?Okay, obviously the person writing this had something odd in mind. I'd better find out what.? That is more likely to teach about the ?__? convention, than to be seen as a mistake by experienced programmers. -- \ ?You don't change the world by placidly finding your bliss ? | `\ you do it by focusing your discontent in productive ways.? | _o__) ?Paul Z. Myers, 2011-08-31 | Ben Finney From srkunze at mail.de Tue Oct 13 22:20:55 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 13 Oct 2015 22:20:55 +0200 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <85eggytup8.fsf@benfinney.id.au> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <85eggytup8.fsf@benfinney.id.au> Message-ID: <561D67A7.5000406@mail.de> On 13.10.2015 21:49, Ben Finney wrote: > What you cite as an advantage is exactly why I'd advise against it. Feel free to do so as do I. > Your example would lead me to ask ?Is this a mistake? ?each? is a normal > name, why isn't it being used?? You as an experienced Python developer. That thread is about students learning programming. Not learning Python. > Whereas ?__? is not a normal name, and like other unusual names in > Python should lead the reader to say ?Okay, obviously the person writing > this had something odd in mind. I'd better find out what.? That is more > likely to teach about the ?__? convention, than to be seen as a mistake > by experienced programmers. Python convention necessary when the student need to write real Python. Best, Sven From breamoreboy at yahoo.co.uk Tue Oct 13 22:21:12 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 13 Oct 2015 21:21:12 +0100 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <561D469B.9020305@mail.de> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> Message-ID: On 13/10/2015 18:59, Sven R. Kunze wrote: > On 13.10.2015 00:21, Ben Finney wrote: >> Since this is quite clearly expressed in existing Python syntax with:: >> >> for __ in range(3): >> forward(100) >> turn_left(120) >> >> the single use case doesn't IMO argue for benefit sufficient to overcome >> the enormous burden of new syntax. > > I still like instead of the cryptic __ or _ > > for each in range(3): > dosomething > > To me, that reads like a normal English statement. > > Best, > Sven No thanks. The current situation has suited the Python community for 25 years give or take, so if it ain't broke, there's no way it's going to get fixed, thank goodness. Can we move on please? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From srkunze at mail.de Tue Oct 13 22:35:25 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 13 Oct 2015 22:35:25 +0200 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> Message-ID: <561D6B0D.8050502@mail.de> On 13.10.2015 22:21, Mark Lawrence wrote: > On 13/10/2015 18:59, Sven R. Kunze wrote: >> I still like instead of the cryptic __ or _ >> >> for each in range(3): >> dosomething >> >> To me, that reads like a normal English statement. >> >> Best, >> Sven > > No thanks. The current situation has suited the Python community for > 25 years give or take, so if it ain't broke, there's no way it's going > to get fixed, thank goodness. Can we move on please? > What exactly do you mean? From breamoreboy at yahoo.co.uk Wed Oct 14 00:19:46 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 13 Oct 2015 23:19:46 +0100 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <561D6B0D.8050502@mail.de> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <561D6B0D.8050502@mail.de> Message-ID: On 13/10/2015 21:35, Sven R. Kunze wrote: > On 13.10.2015 22:21, Mark Lawrence wrote: >> On 13/10/2015 18:59, Sven R. Kunze wrote: >>> I still like instead of the cryptic __ or _ >>> >>> for each in range(3): >>> dosomething >>> >>> To me, that reads like a normal English statement. >>> >>> Best, >>> Sven >> >> No thanks. The current situation has suited the Python community for >> 25 years give or take, so if it ain't broke, there's no way it's going >> to get fixed, thank goodness. Can we move on please? >> > > What exactly do you mean? > Stop pointlessly discussing something that simply is not going to happen. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From rymg19 at gmail.com Wed Oct 14 00:24:17 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 13 Oct 2015 17:24:17 -0500 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <561D6B0D.8050502@mail.de> Message-ID: <272CC295-56A4-42BA-A81D-D6829803C9A4@gmail.com> Like multi-line lambdas, multi-line with statements, null coalescing operators, and all the other topics that come up every other week? On October 13, 2015 5:19:46 PM CDT, Mark Lawrence wrote: >On 13/10/2015 21:35, Sven R. Kunze wrote: >> On 13.10.2015 22:21, Mark Lawrence wrote: >>> On 13/10/2015 18:59, Sven R. Kunze wrote: >>>> I still like instead of the cryptic __ or _ >>>> >>>> for each in range(3): >>>> dosomething >>>> >>>> To me, that reads like a normal English statement. >>>> >>>> Best, >>>> Sven >>> >>> No thanks. The current situation has suited the Python community >for >>> 25 years give or take, so if it ain't broke, there's no way it's >going >>> to get fixed, thank goodness. Can we move on please? >>> >> >> What exactly do you mean? >> > >Stop pointlessly discussing something that simply is not going to >happen. > >-- >My fellow Pythonistas, ask not what our language can do for you, ask >what you can do for our language. > >Mark Lawrence > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 14 02:01:12 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 14 Oct 2015 11:01:12 +1100 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <85eggytup8.fsf@benfinney.id.au> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <85eggytup8.fsf@benfinney.id.au> Message-ID: <20151014000111.GG13813@ando.pearwood.info> On Wed, Oct 14, 2015 at 06:49:39AM +1100, Ben Finney wrote: [...] > Whereas ?__? is not a normal name, and like other unusual names in > Python should lead the reader to say ?Okay, obviously the person writing > this had something odd in mind. I'd better find out what.? That is more > likely to teach about the ?__? convention, than to be seen as a mistake > by experienced programmers. I'd just like to point out that using __ as a loop variable (or any other variable) is incompatible with the IPython convention for __. It is just wrong to talk about "the" __ convention, since there are at least two in common use, and using one necessarily conflicts with using the other. If anyone wants to argue the pros and cons of one convention versus the other, or what unused loop variables ought to be called (x, unused, each, _, __, dontcare, whatever...) please take it to python-list at python.org or comp.lang.python, as it is off-topic here. Thanks, -- Steve From python at lucidity.plus.com Wed Oct 14 02:01:50 2015 From: python at lucidity.plus.com (Erik) Date: Wed, 14 Oct 2015 01:01:50 +0100 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <561D6B0D.8050502@mail.de> Message-ID: <561D9B6E.9020601@lucidity.plus.com> On 13/10/15 23:19, Mark Lawrence wrote: > On 13/10/2015 21:35, Sven R. Kunze wrote: >> On 13.10.2015 22:21, Mark Lawrence wrote: >>> On 13/10/2015 18:59, Sven R. Kunze wrote: >>>> I still like instead of the cryptic __ or _ >>>> >>>> for each in range(3): >>>> dosomething >>>> >>>> To me, that reads like a normal English statement. >>>> >>>> Best, >>>> Sven >>> >>> No thanks. The current situation has suited the Python community for >>> 25 years give or take, so if it ain't broke, there's no way it's going >>> to get fixed, thank goodness. Can we move on please? >>> >> >> What exactly do you mean? > > Stop pointlessly discussing something that simply is not going to happen. Huh? for each in range(3): foo() ... is already valid Python. In what way is that "not going to happen"? This spelling was just suggested as an alternative to the original poster who did not like having to explain the cryptic "___" spelling to students. Same construct, different spelling. E. From python at lucidity.plus.com Wed Oct 14 02:12:05 2015 From: python at lucidity.plus.com (Erik) Date: Wed, 14 Oct 2015 01:12:05 +0100 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <20151014000111.GG13813@ando.pearwood.info> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <85eggytup8.fsf@benfinney.id.au> <20151014000111.GG13813@ando.pearwood.info> Message-ID: <561D9DD5.90406@lucidity.plus.com> On 14/10/15 01:01, Steven D'Aprano wrote: > If anyone wants to argue the pros and cons of one convention versus the > other, or what unused loop variables ought to be called (x, unused, > each, _, __, dontcare, whatever...) please take it to > python-list at python.org or comp.lang.python, as it is off-topic here. The discussion of "what unused loop variables ought to be called" is in direct response to someone suggesting a syntax change to make things easier to teach. To suggest that they spell the loop differently to make it easier to teach whilst not requiring any syntax changes to the language is, I would argue, on-topic as a rebuttal to the change request. However, I agree with you that this particular discussion has probably run its course. E. From tjreedy at udel.edu Wed Oct 14 06:10:41 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Oct 2015 00:10:41 -0400 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: <561D9B6E.9020601@lucidity.plus.com> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <561D6B0D.8050502@mail.de> <561D9B6E.9020601@lucidity.plus.com> Message-ID: On 10/13/2015 8:01 PM, Erik wrote: > On 13/10/15 23:19, Mark Lawrence wrote: >> Stop pointlessly discussing something that simply is not going to happen. > > Huh? > > for each in range(3): > foo() > > ... is already valid Python. In what way is that "not going to happen"? What is not going to happen is that Python will not be changed to actually make a particular identifier a dummy, by not, for instance, letting it be used within the loop. I know you were not proposing that. > This spelling was just suggested as an alternative to the original > poster who did not like having to explain the cryptic "___" spelling to > students. What I think will not and should not happen is that Guido and/or core devs 'bless' in PEP 8 any particular work for teaching beginners. Mark gets a bit over-anxious sometimes, but insofar as he meant 'its futile for us to try to decide on one spelling here', I think he is correct. > Same construct, different spelling. Right, and I think 'each' if better than '_' or '__' for beginners. I recommend 'count' for range() and 'item' for a generic iterable. I also recommend that teachers of beginners experiment with various choices, and share their experiences with each other. Who knows, maybe kids would prefer 'for kangaroo in range(3):', or other personal choices. -- Terry Jan Reedy From stephen at xemacs.org Wed Oct 14 07:28:07 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 14 Oct 2015 14:28:07 +0900 Subject: [Python-ideas] What is Python-ideas for? [meta] [Inspired by: Simpler syntax for basic iterations] In-Reply-To: <272CC295-56A4-42BA-A81D-D6829803C9A4@gmail.com> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <561D6B0D.8050502@mail.de> <272CC295-56A4-42BA-A81D-D6829803C9A4@gmail.com> Message-ID: <22045.59367.919094.178853@turnbull.sk.tsukuba.ac.jp> Ryan Gonzalez writes: Avoid top-posting. It helps prevent posting merely to be contrary. Reordering.... > Mark Lawrence writes: >> Stop pointlessly discussing something that simply is not going to >> happen. > Like multi-line lambdas, multi-line with statements, null > coalescing operators, and all the other topics that come up every > other week? There is a clear qualitative difference here. Mark is right about the "repeat n:" proposal, and your comparison is specious. ***** The rest of my post is an attempt to present my understanding of some aspects of Pythonic design thinking as relevant to discussions on Python-Ideas. As a running example, I try to argue why this thread should have ended with at most two posts in it. (Some might say it shouldn't have started because of PEP 284, but figuring out relevance of several hundred PEPs is a task more suited to scripture scholars than language designers.) Guido admits the benefits of all of the features Ryan mentions, and entertains *new* proposals of syntax to invoke them. That's not because he's a nice guy handing out warm fuzzies to would-be contributors. We currently don't have a good way to do them, where "good" involves both "concise" and "esthetically pleasing, especially to Guido". He really would like to serve the constituencies for those features (and might even warm up to them himself, given suitable syntax). Guido also values the educational use of Python, and when introducing new features he works hard to find syntax (including choice of keyword) that is "intuitive" to both new users of Python and new programmers. If anybody "gets" CP4E, Guido does. Of course there are different ways to get there, but arguing that lack of "repeat n:" is somehow contrary to CP4E is disrespectful. It's a different path to that end. *This proposal* to introduce a special syntax for pure repetition, however, obviously falls afoul of several principles *of Pythonic language design*. On the contrary, the arguments *for* the proposal are "non-Pythonic", which doesn't make them unimportant, much less invalid. It does, however, make them low-value on *this* list, which is for proposed improvements to the Python language. We're not here to discuss language improvements in general.[1] Here's my list: 1. The bar for new keywords is *very* high. I don't think the educational purpose would be well-served by twisting "for" into a knot with "for 4:"; that just doesn't correspond to valid English usage. "while 4:" is a little closer (IMO YMMV), but already has a meaning so it's out anyway due to backward compatibility. I think this proposal needs a new keyword, and "repeat" is excellent for the purpose. That's already a deal-killer for such a specialized construct. 2. We already have a good way to write "repeat 4:". The proposed syntax is just syntactic sugar, and even in its educational context, it's not much sweeter than (proper) use of the existing syntax. Watch one user's lips move as she draws a square: First I draw the top, then I go down the right, then I draw across the bottom, and then I connect the left up to the top. Now compare: repeat 4: forward(10) right(90) versus: for side in ('across top', 'down right', 'across bottom', 'up left'): forward(10) right(90) Which expresses her intent better? Is it really educational to have *this* user to think of a square as an abstract regular polygon that happens to have 4 sides? True, it's easy to cargo- cult the "repeat 4:" idiom the next time she needs a square, but in learning to use "repeat n:", has she learned anything about implementing her own procedures? (This is a variation on the individuality issue that Terry expressed with "kangaroo".) Note: this is a plausible example user, and I am asking whether *she* is served by the new syntax. I don't maintain that she is *the* typical user. I do think that educators should consider her needs as well as the question of how common her type is. While there probably is educational benefit to this change, it's far from clear to me what it is, or whether in fact the student's benefit is anywhere near as large as that to teachers who want to present one "simple" programming construct and move on quickly to another one. 3. The "repeat 4" idiom clearly involves counting (otherwise the program doesn't know when to stop), but it completely conceals the process. That violates "explicit is better than implicit", and furthermore makes the idiom very specialized, dramatically reducing the benefits it can have. It seems to me that the "for ... in" construct gets the balance just right. It exposes both counting as an isomorphism between concrete sets (above, repetitions of a subprocedure corresponding to named sides of a square), and abstractly using numbers merely to count (via "range(n)"). (Of course "range()" also provides a flexible way to denote various concrete sets of numbers.) 4. True, when you really do have an algorithm that simply cares about the number and repeats the same suite independently of the iteration count, exposing an iteration count variable is arguably a wart. But it's very small compared to the power of the "for ... in" idiom. Proposing to address that wart runs into "Special cases aren't special enough to break the rules." Now, all of these principles ("Read my lips: no new keywords!" and the three from the Zen) are pragmatic guidelines, not absolutes. Not even "no keywords". And they're Pythonic principles: they don't always generalize to other languages.[2] But when a proposal infringes so many Pythonic principles so flagrantly (IMHO, each is sufficient reason to deny the proposal by itself, though YMMV), it is indeed a non-starter on this list. Regards, Footnotes: [1] Note that several core Python developers have unsubscribed or muted this list completely, partly because of these off-target threads. [2] I have to admit that I'm thoroughly drunk on the Kool-Aid though, and tend to apply them to other languages, which inevitably fall short. From rymg19 at gmail.com Wed Oct 14 16:14:33 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 14 Oct 2015 09:14:33 -0500 Subject: [Python-ideas] What is Python-ideas for? [meta] [Inspired by: Simpler syntax for basic iterations] In-Reply-To: <22045.59367.919094.178853@turnbull.sk.tsukuba.ac.jp> References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <561D6B0D.8050502@mail.de> <272CC295-56A4-42BA-A81D-D6829803C9A4@gmail.com> <22045.59367.919094.178853@turnbull.sk.tsukuba.ac.jp> Message-ID: <30332C3E-2363-4D1E-86EB-1B05E60D5B9F@gmail.com> On October 14, 2015 12:28:07 AM CDT, "Stephen J. Turnbull" wrote: >Ryan Gonzalez writes: > >Avoid top-posting. It helps prevent posting merely to be >contrary. Reordering.... > > > Mark Lawrence writes: > > >> Stop pointlessly discussing something that simply is not going to > >> happen. > > > Like multi-line lambdas, multi-line with statements, null > > coalescing operators, and all the other topics that come up every > > other week? > >There is a clear qualitative difference here. Mark is right about the >"repeat n:" proposal, and your comparison is specious. > > ***** > >The rest of my post is an attempt to present my understanding of some >aspects of Pythonic design thinking as relevant to discussions on >Python-Ideas. As a running example, I try to argue why this thread >should have ended with at most two posts in it. (Some might say it >shouldn't have started because of PEP 284, but figuring out relevance >of several hundred PEPs is a task more suited to scripture scholars >than language designers.) > ... That was supposed to be a joke. :O >Guido admits the benefits of all of the features Ryan mentions, and >entertains *new* proposals of syntax to invoke them. That's not >because he's a nice guy handing out warm fuzzies to would-be >contributors. We currently don't have a good way to do them, where >"good" involves both "concise" and "esthetically pleasing, especially >to Guido". He really would like to serve the constituencies for those >features (and might even warm up to them himself, given suitable >syntax). > >Guido also values the educational use of Python, and when introducing >new features he works hard to find syntax (including choice of >keyword) that is "intuitive" to both new users of Python and new >programmers. If anybody "gets" CP4E, Guido does. Of course there are >different ways to get there, but arguing that lack of "repeat n:" is >somehow contrary to CP4E is disrespectful. It's a different path to >that end. > >*This proposal* to introduce a special syntax for pure repetition, >however, obviously falls afoul of several principles *of Pythonic >language design*. On the contrary, the arguments *for* the proposal >are "non-Pythonic", which doesn't make them unimportant, much less >invalid. It does, however, make them low-value on *this* list, which >is for proposed improvements to the Python language. We're not here >to discuss language improvements in general.[1] Here's my list: > >1. The bar for new keywords is *very* high. I don't think the > educational purpose would be well-served by twisting "for" into a > knot with "for 4:"; that just doesn't correspond to valid English > usage. "while 4:" is a little closer (IMO YMMV), but already has > a meaning so it's out anyway due to backward compatibility. I > think this proposal needs a new keyword, and "repeat" is excellent > for the purpose. That's already a deal-killer for such a > specialized construct. > >2. We already have a good way to write "repeat 4:". The proposed > syntax is just syntactic sugar, and even in its educational > context, it's not much sweeter than (proper) use of the existing > syntax. Watch one user's lips move as she draws a square: > > First I draw the top, then I go down the right, then I draw > across the bottom, and then I connect the left up to the top. > > Now compare: > > repeat 4: > forward(10) > right(90) > > versus: > > for side in ('across top', > 'down right', > 'across bottom', > 'up left'): > forward(10) > right(90) > > Which expresses her intent better? Is it really educational to > have *this* user to think of a square as an abstract regular > polygon that happens to have 4 sides? True, it's easy to cargo- > cult the "repeat 4:" idiom the next time she needs a square, but > in learning to use "repeat n:", has she learned anything about > implementing her own procedures? (This is a variation on the > individuality issue that Terry expressed with "kangaroo".) > > Note: this is a plausible example user, and I am asking whether > *she* is served by the new syntax. I don't maintain that she is > *the* typical user. I do think that educators should consider her > needs as well as the question of how common her type is. While > there probably is educational benefit to this change, it's far > from clear to me what it is, or whether in fact the student's > benefit is anywhere near as large as that to teachers who want to > present one "simple" programming construct and move on quickly to > another one. > >3. The "repeat 4" idiom clearly involves counting (otherwise the > program doesn't know when to stop), but it completely conceals > the process. That violates "explicit is better than implicit", > and furthermore makes the idiom very specialized, dramatically > reducing the benefits it can have. > > It seems to me that the "for ... in" construct gets the balance > just right. It exposes both counting as an isomorphism between > concrete sets (above, repetitions of a subprocedure corresponding > to named sides of a square), and abstractly using numbers merely > to count (via "range(n)"). (Of course "range()" also provides a > flexible way to denote various concrete sets of numbers.) > >4. True, when you really do have an algorithm that simply cares about > the number and repeats the same suite independently of the > iteration count, exposing an iteration count variable is arguably > a wart. But it's very small compared to the power of the "for > ... in" idiom. Proposing to address that wart runs into "Special > cases aren't special enough to break the rules." > >Now, all of these principles ("Read my lips: no new keywords!" and the >three from the Zen) are pragmatic guidelines, not absolutes. Not even >"no keywords". And they're Pythonic principles: they don't always >generalize to other languages.[2] But when a proposal infringes so >many Pythonic principles so flagrantly (IMHO, each is sufficient >reason to deny the proposal by itself, though YMMV), it is indeed a >non-starter on this list. > >Regards, > > >Footnotes: >[1] Note that several core Python developers have unsubscribed or >muted this list completely, partly because of these off-target >threads. > >[2] I have to admit that I'm thoroughly drunk on the Kool-Aid >though, and tend to apply them to other languages, which inevitably >fall short. -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From tjreedy at udel.edu Thu Oct 15 03:03:22 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Oct 2015 21:03:22 -0400 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: References: Message-ID: <561EFB5A.1080703@udel.edu> On 10/7/2015 2:31 AM, Paddy3118 wrote: > Just saw what the Perl 6 site does to punt the curious to their entries > on the Rosetta Code site here: http://perl6.org/community/rosettacode Perl6 is apparently different from Perl 5 and before to justify adding Perl 6 sections after the Perl sections, with Perl entries re-written. Given the slow development and use of Perl 6, I can understand the Perl 6 people wanting to promote Perl 6 that way. Python 3 is not quite in the same position. > and I thought maybe we could do something similar on Python.org > ? python-ideas is about future version of the language. The pydotorg list discusses the site https://mail.python.org/mailman/listinfo/pydotorg This would be the place to suggest an addition under 'Community'. But if modern Python 3 code is not welcome on the site (see below), I would not want it promoted as containing examples of 'Python' code. > I know that the core development team of Perl 6 have put a lot of effort > into their RC entries, and over the years I have seen evidence of tasks > being used to highlight areas for subsequent improvement in Perl 6. > > I don't /recognise /such a contribution from core development Pythoneers > on RC but members of the Python community have made great efforts in > making Python solutions available for most tasks on RC in a mixture of > Python 2 and 3 The page you reference seems a bit hostile to Python 3. It equates 'Python' with 'Python 2' and calls code that also works with Python 3 'unidiomatic Python', and makes no mention of the possibility of submitting Python 3. > and also showing greater use of our interactive command > line interface/REPL for solutions which other languages may have, Since one cannot do anything in the REPL that one cannot do in a file, except leave out 'print', I do not see the point. Leaving '>>>' and '...' in examples only makes it hard to copy and paste. > but > rarely show unless the task asks specifically for an answer using a REPL. > > *In short:* Python on RC Reading this page, I am not sure how welcome I would be if I came and tried to contribute. Aside from that, I do not see any obvious way to get of list of tasks without a Python solution, or is that an empty set? > has good code examples - why not advertise that fact on Python.org in a > similar way to the Perl 6 site? Because I do not consider code with 'print x' instead of 'print(x)'. especially when that is the only thing keeping the code from also running on Python 3, to be a 'good code example'. Ditto for 2.x-only code (or 3.x-only code, for that matter) that is not labelled as such. On Python list and elsewhere, newbies frequently post 'I copied this code form ... and it does not run.' because they are running it on the wrong interpreter. > *Disclosure: *I should add that I am an administrator, task writer and > one of the Python code contributors to the Rosetta Code site. Please consider making it more Python 3 friendly than it strikes me as being. Or if my impression is wrong, edit the Python page to at least treat Python 3 on an equal footing. Out of curiosity, and to make a point: Does the current tasks include any that test the quality and completeness of unicode implementation? Something simple like the following, involving astral chars. "Construct a string consisting of the unicode characters U+33, U+333, U+11111, U+334, U+11112, U+11113. U+34. Print the decimal codepoint of the third character before the first occurrence of U+11113. (Answer: 69905.) The code should work on any platform the code could run on." 3.x code: s = "\u0033\u0333\U00011111\u0334\U00011112\U00011113" print(ord(s[s.index('\U00011113')-3])) 2.x code: several lines more to accommodate Windows and narrow unicode builds on some linuxes. -- Terry Jan Reedy From abarnert at yahoo.com Thu Oct 15 04:00:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 14 Oct 2015 19:00:15 -0700 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: <561EFB5A.1080703@udel.edu> References: <561EFB5A.1080703@udel.edu> Message-ID: <392862D4-5C29-429B-80D2-EA2B0E4A2BC3@yahoo.com> On Oct 14, 2015, at 18:03, Terry Reedy wrote: > > The page you reference seems a bit hostile to Python 3. It equates 'Python' with 'Python 2' and calls code that also works with Python 3 'unidiomatic Python', and makes no mention of the possibility of submitting Python 3. It also seems to imply that widespread use of reduce is pythonic, apologizing for the unpythonic need to check for and import it in case of Python 3, which makes me wonder how pythonic the code is even for a decade ago? From steve.dower at python.org Thu Oct 15 15:19:27 2015 From: steve.dower at python.org (Steve Dower) Date: Thu, 15 Oct 2015 06:19:27 -0700 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: <392862D4-5C29-429B-80D2-EA2B0E4A2BC3@yahoo.com> References: <561EFB5A.1080703@udel.edu> <392862D4-5C29-429B-80D2-EA2B0E4A2BC3@yahoo.com> Message-ID: FWIW, that comment is from five years ago and was written by someone who claims to have been using Python far longer than that (and has enough contributions for that to be believable). It reads more to me like a preemptive apology to "The Haters" than a dismissal of Python 3. I'm sure any contributor to the site can easily reword it to better reflect that not all examples are as terse as they could be if they avoided the 2/3 straddle. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Andrew Barnert via Python-ideas" Sent: ?10/?14/?2015 19:03 To: "Terry Reedy" Cc: "python-ideas at python.org" Subject: Re: [Python-ideas] Rosetta Code on Python.org site? On Oct 14, 2015, at 18:03, Terry Reedy wrote: > > The page you reference seems a bit hostile to Python 3. It equates 'Python' with 'Python 2' and calls code that also works with Python 3 'unidiomatic Python', and makes no mention of the possibility of submitting Python 3. It also seems to imply that widespread use of reduce is pythonic, apologizing for the unpythonic need to check for and import it in case of Python 3, which makes me wonder how pythonic the code is even for a decade ago? _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason at schwerberg.com Thu Oct 15 19:37:47 2015 From: jason at schwerberg.com (Jason Schwerberg) Date: Thu, 15 Oct 2015 10:37:47 -0700 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: References: <561EFB5A.1080703@udel.edu> <392862D4-5C29-429B-80D2-EA2B0E4A2BC3@yahoo.com> Message-ID: <561FE46B.2070906@schwerberg.com> At risk of completely going off topic from the mission of Python-Ideas, I'd almost rather see a delineation between Python 2 and Python 3, like RC has done with Perl 5 (http://rosettacode.org/wiki/Category:Perl) and Perl 6 (http://rosettacode.org/wiki/Category:Perl_6). From RC: > Though it resembles previous versions of Perl to no small degree, Perl 6 is substantially a new language; by design, it isn't backwards-compatible with Perl 5. And from https://wiki.python.org/moin/Python2orPython3 > Guido van Rossum (the original creator of the Python language) decided to clean up Python 2.x properly, with less regard for backwards compatibility than is the case for new releases in the 2.x range. While there are many ways to make 'good' code, compatible with both Python 2 and 3, I think that the most terse, efficient code would be better demonstrated by segregating Python 2.x from Python 3.x. On 10/15/2015 6:19 AM, Steve Dower wrote: > FWIW, that comment is from five years ago and was written by someone > who claims to have been using Python far longer than that (and has > enough contributions for that to be believable). It reads more to me > like a preemptive apology to "The Haters" than a dismissal of Python 3. > > I'm sure any contributor to the site can easily reword it to better > reflect that not all examples are as terse as they could be if they > avoided the 2/3 straddle. > > Cheers, > Steve > > Top-posted from my Windows Phone > ------------------------------------------------------------------------ > From: Andrew Barnert via Python-ideas > Sent: ?10/?14/?2015 19:03 > To: Terry Reedy > Cc: python-ideas at python.org > Subject: Re: [Python-ideas] Rosetta Code on Python.org site? > > On Oct 14, 2015, at 18:03, Terry Reedy wrote: > > > > The page you reference seems a bit hostile to Python 3. It equates > 'Python' with 'Python 2' and calls code that also works with Python 3 > 'unidiomatic Python', and makes no mention of the possibility of > submitting Python 3. > > It also seems to imply that widespread use of reduce is pythonic, > apologizing for the unpythonic need to check for and import it in case > of Python 3, which makes me wonder how pythonic the code is even for a > decade ago? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Oct 15 23:14:16 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 15 Oct 2015 14:14:16 -0700 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: <561FE46B.2070906@schwerberg.com> References: <561EFB5A.1080703@udel.edu> <392862D4-5C29-429B-80D2-EA2B0E4A2BC3@yahoo.com> <561FE46B.2070906@schwerberg.com> Message-ID: On Thu, Oct 15, 2015 at 10:37 AM, Jason Schwerberg wrote: > At risk of completely going off topic from the mission of Python-Ideas, > I'd almost rather see a delineation between Python 2 and Python 3 > > While there are many ways to make 'good' code, compatible with both Python > 2 and 3, I think that the most terse, efficient code would be better > demonstrated by segregating Python 2.x from Python 3.x. > makes sense to me. RC is a wiki, yes? So it looks like this: http://rosettacode.org/wiki/Rosetta_Code:Add_a_Language Is what you need to do... Or would there really be strong objections from the community to having a Python3 section on the site? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokoproject at gmail.com Thu Oct 15 23:25:16 2015 From: gokoproject at gmail.com (John Wong) Date: Thu, 15 Oct 2015 17:25:16 -0400 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: References: Message-ID: On Wed, Oct 7, 2015 at 2:31 AM, Paddy3118 wrote: > Just saw what the Perl 6 site does to punt the curious to their entries > on the Rosetta Code site here: http://perl6.org/community/rosettacode and > I thought maybe we could do something similar on Python.org > ? > > Problem with rosettacode is they can end up like stackoverflow answers that some are good and some are bad. Some will view that as Bible and some will constantly make "suggestions" why X is better. But in the spirit of playground and show case, why not make this a personal project and have people commit to a public repo such as GitHub? Everyone will come up with a different solution. Take BFS and DFS as examples, there are at least two kinds of popular ways to implement that in Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokoproject at gmail.com Thu Oct 15 23:29:57 2015 From: gokoproject at gmail.com (John Wong) Date: Thu, 15 Oct 2015 17:29:57 -0400 Subject: [Python-ideas] Simpler syntax for basic iterations In-Reply-To: References: <22042.5180.435279.647018@turnbull.sk.tsukuba.ac.jp> <85mvvnu3ro.fsf@benfinney.id.au> <561D469B.9020305@mail.de> <561D6B0D.8050502@mail.de> <561D9B6E.9020601@lucidity.plus.com> Message-ID: On Wed, Oct 14, 2015 at 12:10 AM, Terry Reedy wrote: > > This spelling was just suggested as an alternative to the original >> poster who did not like having to explain the cryptic "___" spelling to >> students. >> > > What I think will not and should not happen is that Guido and/or core devs > 'bless' in PEP 8 any particular work for teaching beginners. Mark gets a > bit over-anxious sometimes, but insofar as he meant 'its futile for us to > try to decide on one spelling here', I think he is correct. > > Same construct, different spelling. >> > > Right, and I think 'each' if better than '_' or '__' for beginners. I > recommend 'count' for range() and 'item' for a generic iterable. I also > recommend that teachers of beginners experiment with various choices, and > share their experiences with each other. Who knows, maybe kids would prefer > 'for kangaroo in range(3):', or other personal choices. > > If a student is asking _ then it should be explained. Soon or later the students will probably see _ in other source code. Whether it is a 4-year old child or a 18 year-old college freshman. I wouldn't expect my nephew to understand everything I write, but they will grow up and have the "ah" moment. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Oct 16 02:02:42 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Oct 2015 11:02:42 +1100 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: References: <561EFB5A.1080703@udel.edu> <392862D4-5C29-429B-80D2-EA2B0E4A2BC3@yahoo.com> <561FE46B.2070906@schwerberg.com> Message-ID: <20151016000241.GA11980@ando.pearwood.info> On Thu, Oct 15, 2015 at 02:14:16PM -0700, Chris Barker wrote: > On Thu, Oct 15, 2015 at 10:37 AM, Jason Schwerberg > wrote: > > > At risk of completely going off topic from the mission of Python-Ideas, > > I'd almost rather see a delineation between Python 2 and Python 3 > > > While there are many ways to make 'good' code, compatible with both Python > > 2 and 3, I think that the most terse, efficient code would be better > > demonstrated by segregating Python 2.x from Python 3.x. > > makes sense to me. RC is a wiki, yes? So it looks like this: > > http://rosettacode.org/wiki/Rosetta_Code:Add_a_Language > > Is what you need to do... > > Or would there really be strong objections from the community to having a > Python3 section on the site? I would object to adding Python 3 as a different language. It is not, and we should be emphasising the similarities, not the differences. Migration to Python 3 is slow enough without giving more ammunition ("it's a completely different language, like Perl 6 is to Perl 5") to the foot-draggers and haters. Adding Python 3 as a different language would be a strategic error, because most code snippets will be identical except for possibly print and (raw_)input. That's going to discourage people from adding a Python 3 version that is all but identical to the Python 2 version, which will increase the perception that nobody is using Python 3. ("There are hundreds of Python 2 examples, and only a dozen Python 3 examples.") In my opinion, as far as Rosetta Code and similar sites go, all it takes to manage the Python 2/3 transition is a single comment at the start of the code snippet listing the version(s) targeted. I would consider that the minimum needed. Actually providing hybrid 2+3 code, or side-by-side versions, would be acceptible too, but that's more work. I just looked at a small sample of arbitrary examples, and found that three out of the four already support Python 3, and the one that didn't just needed parens added to print to work. So I don't think there's actually a problem here that needs fixing. -- Steven From tjreedy at udel.edu Fri Oct 16 02:59:41 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 15 Oct 2015 20:59:41 -0400 Subject: [Python-ideas] Rosetta Code on Python.org site? In-Reply-To: <20151016000241.GA11980@ando.pearwood.info> References: <561EFB5A.1080703@udel.edu> <392862D4-5C29-429B-80D2-EA2B0E4A2BC3@yahoo.com> <561FE46B.2070906@schwerberg.com> <20151016000241.GA11980@ando.pearwood.info> Message-ID: On 10/15/2015 8:02 PM, Steven D'Aprano wrote: > On Thu, Oct 15, 2015 at 02:14:16PM -0700, Chris Barker wrote: >> On Thu, Oct 15, 2015 at 10:37 AM, Jason Schwerberg >> wrote: >> >>> At risk of completely going off topic from the mission of Python-Ideas, >>> I'd almost rather see a delineation between Python 2 and Python 3 >> >>> While there are many ways to make 'good' code, compatible with both Python >>> 2 and 3, I think that the most terse, efficient code would be better >>> demonstrated by segregating Python 2.x from Python 3.x. >> >> makes sense to me. RC is a wiki, yes? So it looks like this: >> >> http://rosettacode.org/wiki/Rosetta_Code:Add_a_Language >> >> Is what you need to do... >> >> Or would there really be strong objections from the community to having a >> Python3 section on the site? Yes, Python 3 *is* Python. Python has always been dynamic in its definition, as well as its execution. 'Python 2' is a sequence of older, frozen dialects. Note that 2.7 code may not run in older versions. And that 2.0 code, for instance, may not work in some 2.x and after. Python 2 sections could be added for 2.x-only code, but for most of the tasks, solutions that run on 3.x and some subset of 2.x are easily possible. When not, the format allow multiple solutions within a language section. For example, FizzBuzz/Python has a both 'print 100 lines' for-loop solution and and infinite itertools-based iterator solution. > I would object to adding Python 3 as a different language. It is not, > and we should be emphasising the similarities, not the differences. > Migration to Python 3 is slow enough without giving more ammunition > ("it's a completely different language, like Perl 6 is to Perl 5") to > the foot-draggers and haters. This is why I said the Perl6/5- was not the example to follow. > Adding Python 3 as a different language would be a strategic error, > because most code snippets will be identical except for possibly > print and (raw_)input. That's going to discourage people from adding a > Python 3 version that is all but identical to the Python 2 version, > which will increase the perception that nobody is using Python 3. > ("There are hundreds of Python 2 examples, and only a dozen Python 3 > examples.") > > In my opinion, as far as Rosetta Code and similar sites go, all it takes > to manage the Python 2/3 transition is a single comment at the start of > the code snippet listing the version(s) targeted. I would consider that > the minimum needed. Actually providing hybrid 2+3 code, or side-by-side > versions, would be acceptible too, but that's more work. > > I just looked at a small sample of arbitrary examples, and found that > three out of the four already support Python 3, and the one that didn't > just needed parens added to print to work. > > So I don't think there's actually a problem here that needs fixing. Except for version labeling and blessing rather than dissing 3.x or 2&3 solutions. -- Terry Jan Reedy From ian at feete.org Fri Oct 16 12:36:42 2015 From: ian at feete.org (Ian Foote) Date: Fri, 16 Oct 2015 11:36:42 +0100 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict Message-ID: <5620D33A.3050709@feete.org> I recently wanted to use an OrderedCounter and an OrderedDefaultDict. After a bit of googling I discovered that OrderedCounter is quite easy to implement: from collections import Counter, OrderedDict class OrderedCounter(Counter, OrderedDict): 'Counter that remembers the order elements are first seen' def __repr__(self): return '%s(%r)' % (self.__class__.__name__, OrderedDict(self)) def __reduce__(self): return self.__class__, (OrderedDict(self),) from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ Unfortunately an OrderedDefaultDict did not seem so easy. I did find http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict-in-python which suggests: |fromcollections importOrderedDict,CallableclassDefaultOrderedDict(OrderedDict):# Source: http://stackoverflow.com/a/6190500/562769def__init__(self,default_factory=None,*a,**kw):if(default_factory isnotNoneandnotisinstance(default_factory,Callable)):raiseTypeError('first argument must be callable')OrderedDict.__init__(self,*a,**kw)self.default_factory =default_factory def__getitem__(self,key):try:returnOrderedDict.__getitem__(self,key)exceptKeyError:returnself.__missing__(key)def__missing__(self,key):ifself.default_factory isNone:raiseKeyError(key)self[key]=value =self.default_factory()returnvalue def__reduce__(self):ifself.default_factory isNone:args =tuple()else:args =self.default_factory,returntype(self),args,None,None,self.items()defcopy(self):returnself.__copy__()def__copy__(self):returntype(self)(self.default_factory,self)def__deepcopy__(self,memo):importcopy returntype(self)(self.default_factory,copy.deepcopy(self.items()))def__repr__(self):return'OrderedDefaultDict(%s, %s)'%(self.default_factory,OrderedDict.__repr__(self))| This seems to me both easy to get wrong and hard to discover. It also risks getting out of sync with updates to collections. I'm wondering if this is sufficient justification to add OrderedCounter and OrderedDict to collections, either directly or as recipes. Thanks, Ian F -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Oct 17 03:39:02 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 16 Oct 2015 18:39:02 -0700 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <5620D33A.3050709@feete.org> References: <5620D33A.3050709@feete.org> Message-ID: <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> On Oct 16, 2015, at 03:36, Ian Foote wrote: > > I recently wanted to use an OrderedCounter and an OrderedDefaultDict. After a bit of googling I discovered that OrderedCounter is quite easy to implement: This is already in the docs for OrderedDict, so it shouldn't have taken any googling. > from collections import Counter, OrderedDict > > class OrderedCounter(Counter, OrderedDict): > 'Counter that remembers the order elements are first seen' > def __repr__(self): > return '%s(%r)' % (self.__class__.__name__, > OrderedDict(self)) > def __reduce__(self): > return self.__class__, (OrderedDict(self),) > > from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ > > Unfortunately an OrderedDefaultDict did not seem so easy. I did find http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict-in-python which suggests: Most of the trickiness here is handling None as a factory, which is only useful for subclasses that implement a custom __missing__. To make something that works like you'd expect except for that part is a lot simpler. And I'm not sure a docs recipe that's mostly code that's unnecessary for most uses, hiding the important and simple part, would be a great idea. > from collections import OrderedDict, Callable > > class DefaultOrderedDict(OrderedDict): > # Source: http://stackoverflow.com/a/6190500/562769 > def __init__(self, default_factory=None, *a, **kw): > if (default_factory is not None and > not isinstance(default_factory, Callable)): > raise TypeError('first argument must be callable') > OrderedDict.__init__(self, *a, **kw) > self.default_factory = default_factory > > def __getitem__(self, key): > try: > return OrderedDict.__getitem__(self, key) > except KeyError: > return self.__missing__(key) > > def __missing__(self, key): > if self.default_factory is None: > raise KeyError(key) > self[key] = value = self.default_factory() > return value > > def __reduce__(self): > if self.default_factory is None: > args = tuple() > else: > args = self.default_factory, > return type(self), args, None, None, self.items() > > def copy(self): > return self.__copy__() > > def __copy__(self): > return type(self)(self.default_factory, self) > > def __deepcopy__(self, memo): > import copy > return type(self)(self.default_factory, > copy.deepcopy(self.items())) > > def __repr__(self): > return 'OrderedDefaultDict(%s, %s)' % (self.default_factory, > OrderedDict.__repr__(self)) > > This seems to me both easy to get wrong and hard to discover. It also risks getting out of sync with updates to collections. > I'm wondering if this is sufficient justification to add OrderedCounter and OrderedDict to collections, either directly or as recipes. Since one of them is already a recipe and the other one would probably not be useful as one, I don't think that's a great idea. Adding OrderedDefaultDict to the module itself might be useful. Or, alternatively, put it in PyPI or ActiveState and add a link to it from the docs? (I'm pretty sure there are already implementations in both places, actually. Also, it would be nice to be able to point at a third-party implementation that works in 2.6+/3.2+ or whatever even if it only appears in the 3.6 docs.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Oct 17 03:57:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 16 Oct 2015 18:57:21 -0700 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> Message-ID: Sorry, sent too soon? Sent from my iPhone > On Oct 16, 2015, at 18:39, Andrew Barnert via Python-ideas wrote: > >> On Oct 16, 2015, at 03:36, Ian Foote wrote: >> >> I recently wanted to use an OrderedCounter and an OrderedDefaultDict. After a bit of googling I discovered that OrderedCounter is quite easy to implement: > > This is already in the docs for OrderedDict, so it shouldn't have taken any googling. >> from collections import Counter, OrderedDict >> >> class OrderedCounter(Counter, OrderedDict): >> 'Counter that remembers the order elements are first seen' >> def __repr__(self): >> return '%s(%r)' % (self.__class__.__name__, >> OrderedDict(self)) >> def __reduce__(self): >> return self.__class__, (OrderedDict(self),) >> >> from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ >> >> Unfortunately an OrderedDefaultDict did not seem so easy. I did find http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict-in-python which suggests: > > Most of the trickiness here is handling None as a factory, which is only useful for subclasses that implement a custom __missing__ ? or that requires two-stage initialization, where it can initialize the base with None and then assign the factory later. Still not all that common, but it is a separate reason for the extra complexity besides a custom __missing__ that doesn't need the factory. > To make something that works like you'd expect except for that part is a lot simpler. Off the top of my head (untested, since I'm on my phone): class DefaultOrderedDict(OrderedDict): def __init__(self, default_factory, *a, **kw): super().__init__(*a, **kw) self.default_factory = default_factory def __getitem__(self, key): try: return super().__getitem__(key) except KeyError: return self.__missing__(key) def __missing__(self, key): self[key] = value = self.default_factory() return value def __repr__(self): return '{}({}, {})'.format(type(self).__name__, self.default_factory, dict.__repr__(self)) In fact, if you aren't worried about subclassing, or about being able to directly call or pass around a bound__missing__, you can make this even simpler. I may be wrong about this being picklable/copyable without any extra help; if so, it still doesn't need the whole complicated "treat a None factory as if it doesn't exist" part. Also, I think the SO recipe works in Python 2.4+, or whichever first added OrderedDict, while this requires 3.1 (although it should be obvious how to backport it, make sure that OrderedDict isn't an old-style class?). Anyway, other than those issues, it seems pretty obvious, and I don't think there's any part of that functionality that's easy to get wrong. Handling a None factory in __missing__ and pickle/copy is easy to get wrong, but if you aren't going to subclass this and therefore aren't going to try to build that part, you're not going to build it wrong. > And I'm not sure a docs recipe that's mostly code that's unnecessary for most uses, hiding the important and simple part, would be a great idea. >> from collections import OrderedDict, Callable >> >> class DefaultOrderedDict(OrderedDict): >> # Source: http://stackoverflow.com/a/6190500/562769 >> def __init__(self, default_factory=None, *a, **kw): >> if (default_factory is not None and >> not isinstance(default_factory, Callable)): >> raise TypeError('first argument must be callable') >> OrderedDict.__init__(self, *a, **kw) >> self.default_factory = default_factory >> >> def __getitem__(self, key): >> try: >> return OrderedDict.__getitem__(self, key) >> except KeyError: >> return self.__missing__(key) >> >> def __missing__(self, key): >> if self.default_factory is None: >> raise KeyError(key) >> self[key] = value = self.default_factory() >> return value >> >> def __reduce__(self): >> if self.default_factory is None: >> args = tuple() >> else: >> args = self.default_factory, >> return type(self), args, None, None, self.items() >> >> def copy(self): >> return self.__copy__() >> >> def __copy__(self): >> return type(self)(self.default_factory, self) >> >> def __deepcopy__(self, memo): >> import copy >> return type(self)(self.default_factory, >> copy.deepcopy(self.items())) >> >> def __repr__(self): >> return 'OrderedDefaultDict(%s, %s)' % (self.default_factory, >> OrderedDict.__repr__(self)) >> >> This seems to me both easy to get wrong and hard to discover. It also risks getting out of sync with updates to collections. >> I'm wondering if this is sufficient justification to add OrderedCounter and OrderedDict to collections, either directly or as recipes. > > Since one of them is already a recipe and the other one would probably not be useful as one, I don't think that's a great idea. > > Adding OrderedDefaultDict to the module itself might be useful. Or, alternatively, put it in PyPI or ActiveState and add a link to it from the docs? (I'm pretty sure there are already implementations in both places, actually. Also, it would be nice to be able to point at a third-party implementation that works in 2.6+/3.2+ or whatever even if it only appears in the 3.6 docs.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Oct 17 04:08:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 16 Oct 2015 19:08:15 -0700 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> Message-ID: <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> Actually, forget all that; it's even simpler. At least in recent 3.x, the only thing wrong with inheriting from both types, assuming you put OrderedDict first, is the __init__ signature. So: class OrderedDefaultDict(OrderedDict, defaultdict): def __init__(self, default_factory=None, *a, **kw): OrderedDict.__init__(self, *a, **kw) self.default_factory = default_factory More importantly, because __missing__ support is built into dict, despite the confusing docs for defaultdict, you don't really need defaultdict at all here: class OrderedDefaultDict(OrderedDict): def __init__(self, default_factory=None, *a, **kw): OrderedDict.__init__(self, *a, **kw) self.default_factory = default_factory def __missing__(self, key): self[key] = value = default_factory() return value And either of these should work with 2.5+ (according to https://docs.python.org/2/library/stdtypes.html#dict that's when dict.__missing__ was added). Sent from my iPhone > On Oct 16, 2015, at 18:57, Andrew Barnert wrote: > > Sorry, sent too soon? > > Sent from my iPhone > >> On Oct 16, 2015, at 18:39, Andrew Barnert via Python-ideas wrote: >> >>> On Oct 16, 2015, at 03:36, Ian Foote wrote: >>> >>> I recently wanted to use an OrderedCounter and an OrderedDefaultDict. After a bit of googling I discovered that OrderedCounter is quite easy to implement: >> >> This is already in the docs for OrderedDict, so it shouldn't have taken any googling. >>> from collections import Counter, OrderedDict >>> >>> class OrderedCounter(Counter, OrderedDict): >>> 'Counter that remembers the order elements are first seen' >>> def __repr__(self): >>> return '%s(%r)' % (self.__class__.__name__, >>> OrderedDict(self)) >>> def __reduce__(self): >>> return self.__class__, (OrderedDict(self),) >>> >>> from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ >>> >>> Unfortunately an OrderedDefaultDict did not seem so easy. I did find http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict-in-python which suggests: >> >> Most of the trickiness here is handling None as a factory, which is only useful for subclasses that implement a custom __missing__ > > ? or that requires two-stage initialization, where it can initialize the base with None and then assign the factory later. Still not all that common, but it is a separate reason for the extra complexity besides a custom __missing__ that doesn't need the factory. > >> To make something that works like you'd expect except for that part is a lot simpler. > > Off the top of my head (untested, since I'm on my phone): > > class DefaultOrderedDict(OrderedDict): > def __init__(self, default_factory, *a, **kw): > super().__init__(*a, **kw) > self.default_factory = default_factory > def __getitem__(self, key): > try: > return super().__getitem__(key) > except KeyError: > return self.__missing__(key) > def __missing__(self, key): > self[key] = value = self.default_factory() > return value > def __repr__(self): > return '{}({}, {})'.format(type(self).__name__, self.default_factory, dict.__repr__(self)) > > In fact, if you aren't worried about subclassing, or about being able to directly call or pass around a bound__missing__, you can make this even simpler. > > I may be wrong about this being picklable/copyable without any extra help; if so, it still doesn't need the whole complicated "treat a None factory as if it doesn't exist" part. > > Also, I think the SO recipe works in Python 2.4+, or whichever first added OrderedDict, while this requires 3.1 (although it should be obvious how to backport it, make sure that OrderedDict isn't an old-style class?). > > Anyway, other than those issues, it seems pretty obvious, and I don't think there's any part of that functionality that's easy to get wrong. Handling a None factory in __missing__ and pickle/copy is easy to get wrong, but if you aren't going to subclass this and therefore aren't going to try to build that part, you're not going to build it wrong. > >> And I'm not sure a docs recipe that's mostly code that's unnecessary for most uses, hiding the important and simple part, would be a great idea. >>> from collections import OrderedDict, Callable >>> >>> class DefaultOrderedDict(OrderedDict): >>> # Source: http://stackoverflow.com/a/6190500/562769 >>> def __init__(self, default_factory=None, *a, **kw): >>> if (default_factory is not None and >>> not isinstance(default_factory, Callable)): >>> raise TypeError('first argument must be callable') >>> OrderedDict.__init__(self, *a, **kw) >>> self.default_factory = default_factory >>> >>> def __getitem__(self, key): >>> try: >>> return OrderedDict.__getitem__(self, key) >>> except KeyError: >>> return self.__missing__(key) >>> >>> def __missing__(self, key): >>> if self.default_factory is None: >>> raise KeyError(key) >>> self[key] = value = self.default_factory() >>> return value >>> >>> def __reduce__(self): >>> if self.default_factory is None: >>> args = tuple() >>> else: >>> args = self.default_factory, >>> return type(self), args, None, None, self.items() >>> >>> def copy(self): >>> return self.__copy__() >>> >>> def __copy__(self): >>> return type(self)(self.default_factory, self) >>> >>> def __deepcopy__(self, memo): >>> import copy >>> return type(self)(self.default_factory, >>> copy.deepcopy(self.items())) >>> >>> def __repr__(self): >>> return 'OrderedDefaultDict(%s, %s)' % (self.default_factory, >>> OrderedDict.__repr__(self)) >>> >>> This seems to me both easy to get wrong and hard to discover. It also risks getting out of sync with updates to collections. >>> I'm wondering if this is sufficient justification to add OrderedCounter and OrderedDict to collections, either directly or as recipes. >> >> Since one of them is already a recipe and the other one would probably not be useful as one, I don't think that's a great idea. >> >> Adding OrderedDefaultDict to the module itself might be useful. Or, alternatively, put it in PyPI or ActiveState and add a link to it from the docs? (I'm pretty sure there are already implementations in both places, actually. Also, it would be nice to be able to point at a third-party implementation that works in 2.6+/3.2+ or whatever even if it only appears in the 3.6 docs.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Sun Oct 18 00:17:47 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Sat, 17 Oct 2015 18:17:47 -0400 Subject: [Python-ideas] Discussion about a 'xor' keyword? Message-ID: You've probably read this title with skepticism. I know I would. As the title says. I am curious as to engage in the discussion, and see what everyone's thoughts are on it. I know this has been discussed in the past several years ago ( https://mail.python.org/pipermail/python-3000/2007-December/011560.html ) and on various websites already. I've found myself writing code that would be simpler with a xor keyword or similar syntax, and this is why I'm suggesting this. Here are my thoughts: Since 'a or b' and 'a and b' return either a or b, what should 'a xor b' return? I have a few ideas, and no particular preference: To be consistent with other binary operators, 'xor' should return one of its operands, IMO. So 'a xor b' should be a if a is True and b is False, and b if b is True and a is False. That much makes sense so far (to me). What I'm a bit less sure about is what to return in the case that both are True, or both False. We already have a per-type definition of xor, and if we want to have only the boolean xor, we can easily do 'bool(a) ^ bool(b)' and so I feel this discussion should disregard the possibility of returning only a boolean value; we can already do that in a way that a keyword wouldn't help. My thoughts on this would be to return either a or b if only one of them is True in a boolean context, else return None. None is a fairly common name already, and we could use it to say that the exclusive-or check returned no value (just like a function call). I don't think adding a new builtin name would be the way to go - adding a new keyword is already a big breaking change. If one of the two values is None already... well then I guess that's something to be fixed in the code? That's one of the things that should probably be discussed, should the people on this list be favorable to the idea. I'm expecting there'll be some sort of BDFL pronouncement at some point, but I still want to have the discussion. If it gets rejected, I would at least be happy if some discussion took place to talk about it. -Emanuel Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Oct 18 00:49:42 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 18 Oct 2015 01:49:42 +0300 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: Message-ID: On 18.10.15 01:17, Emanuel Barry wrote: > My thoughts on this would be to return either a or b if only one of them > is True in a boolean context, else return None. None is a fairly common > name already, and we could use it to say that the exclusive-or check > returned no value (just like a function call). I don't think adding a > new builtin name would be the way to go - adding a new keyword is > already a big breaking change. If one of the two values is None > already... well then I guess that's something to be fixed in the code? > That's one of the things that should probably be discussed, should the > people on this list be favorable to the idea. 'and' and 'or' are short-circuit operators. But 'xor' needs to know values of both operands. Therefore no need in the xor operator, it can be a function. def xor(a, b): if a: if b: return None else: return a else: if b: return b else: return None But I suppose that in most cases boolean result is enough, and it can be written as bool(a) != bool(b). From emile at fenx.com Sun Oct 18 00:50:35 2015 From: emile at fenx.com (Emile van Sebille) Date: Sat, 17 Oct 2015 15:50:35 -0700 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: Message-ID: Do you have additional arguments that weren't covered in the PEPs discussing the addition of an XOR operator? Emile On 10/17/2015 3:17 PM, Emanuel Barry wrote: > You've probably read this title with skepticism. I know I would. > > As the title says. I am curious as to engage in the discussion, and see > what everyone's thoughts are on it. I know this has been discussed in > the past several years ago ( > https://mail.python.org/pipermail/python-3000/2007-December/011560.html ) and > on various websites already. I've found myself writing code that would > be simpler with a xor keyword or similar syntax, and this is why I'm > suggesting this. > > Here are my thoughts: > > Since 'a or b' and 'a and b' return either a or b, what should 'a xor b' > return? I have a few ideas, and no particular preference: > > To be consistent with other binary operators, 'xor' should return one of > its operands, IMO. So 'a xor b' should be a if a is True and b is False, > and b if b is True and a is False. That much makes sense so far (to me). > What I'm a bit less sure about is what to return in the case that both > are True, or both False. > > We already have a per-type definition of xor, and if we want to have > only the boolean xor, we can easily do 'bool(a) ^ bool(b)' and so I feel > this discussion should disregard the possibility of returning only a > boolean value; we can already do that in a way that a keyword wouldn't help. > > My thoughts on this would be to return either a or b if only one of them > is True in a boolean context, else return None. None is a fairly common > name already, and we could use it to say that the exclusive-or check > returned no value (just like a function call). I don't think adding a > new builtin name would be the way to go - adding a new keyword is > already a big breaking change. If one of the two values is None > already... well then I guess that's something to be fixed in the code? > That's one of the things that should probably be discussed, should the > people on this list be favorable to the idea. > > I'm expecting there'll be some sort of BDFL pronouncement at some point, > but I still want to have the discussion. If it gets rejected, I would at > least be happy if some discussion took place to talk about it. > > -Emanuel Barry > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From vgr255 at live.ca Sun Oct 18 00:59:23 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Sat, 17 Oct 2015 18:59:23 -0400 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: , Message-ID: > To: python-ideas at python.org > From: storchaka at gmail.com > Date: Sun, 18 Oct 2015 01:49:42 +0300 > Subject: Re: [Python-ideas] Discussion about a 'xor' keyword? > > On 18.10.15 01:17, Emanuel Barry wrote: > > My thoughts on this would be to return either a or b if only one of them > > is True in a boolean context, else return None. None is a fairly common > > name already, and we could use it to say that the exclusive-or check > > returned no value (just like a function call). I don't think adding a > > new builtin name would be the way to go - adding a new keyword is > > already a big breaking change. If one of the two values is None > > already... well then I guess that's something to be fixed in the code? > > That's one of the things that should probably be discussed, should the > > people on this list be favorable to the idea. > > 'and' and 'or' are short-circuit operators. But 'xor' needs to know > values of both operands. Therefore no need in the xor operator, it can > be a function. > > def xor(a, b): > if a: > if b: > return None > else: > return a > else: > if b: > return b > else: > return None > > But I suppose that in most cases boolean result is enough, and it can be > written as bool(a) != bool(b). I must admit I forgot about the short-circuit feature of the 'and' and 'or' operands while I was writing this. I agree it could be a function (I already use one for most cases), but was mostly suggesting the idea (rather than stating it as a hard necessity). >Do you have additional arguments that weren't covered in the PEPs >discussing the addition of an XOR operator?>Emile I must admit I did not see such PEP (I still don't, unless I'm not looking right). My bad in that case. -Emanuel Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Sun Oct 18 01:28:01 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 18 Oct 2015 00:28:01 +0100 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: Message-ID: On 17/10/2015 23:50, Emile van Sebille wrote: > Do you have additional arguments that weren't covered in the PEPs > discussing the addition of an XOR operator? > > Emile > The only one I can find that refers specifically to xor is PEP 225 -- Elementwise/Objectwise Operators[1]. Which others have I missed? [1] https://www.python.org/dev/peps/pep-0225/ > > On 10/17/2015 3:17 PM, Emanuel Barry wrote: >> You've probably read this title with skepticism. I know I would. >> >> As the title says. I am curious as to engage in the discussion, and see >> what everyone's thoughts are on it. I know this has been discussed in >> the past several years ago ( >> https://mail.python.org/pipermail/python-3000/2007-December/011560.html ) >> and >> on various websites already. I've found myself writing code that would >> be simpler with a xor keyword or similar syntax, and this is why I'm >> suggesting this. >> >> Here are my thoughts: >> >> Since 'a or b' and 'a and b' return either a or b, what should 'a xor b' >> return? I have a few ideas, and no particular preference: >> >> To be consistent with other binary operators, 'xor' should return one of >> its operands, IMO. So 'a xor b' should be a if a is True and b is False, >> and b if b is True and a is False. That much makes sense so far (to me). >> What I'm a bit less sure about is what to return in the case that both >> are True, or both False. >> >> We already have a per-type definition of xor, and if we want to have >> only the boolean xor, we can easily do 'bool(a) ^ bool(b)' and so I feel >> this discussion should disregard the possibility of returning only a >> boolean value; we can already do that in a way that a keyword wouldn't >> help. >> >> My thoughts on this would be to return either a or b if only one of them >> is True in a boolean context, else return None. None is a fairly common >> name already, and we could use it to say that the exclusive-or check >> returned no value (just like a function call). I don't think adding a >> new builtin name would be the way to go - adding a new keyword is >> already a big breaking change. If one of the two values is None >> already... well then I guess that's something to be fixed in the code? >> That's one of the things that should probably be discussed, should the >> people on this list be favorable to the idea. >> >> I'm expecting there'll be some sort of BDFL pronouncement at some point, >> but I still want to have the discussion. If it gets rejected, I would at >> least be happy if some discussion took place to talk about it. >> >> -Emanuel Barry -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From vgr255 at live.ca Sun Oct 18 01:53:22 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Sat, 17 Oct 2015 19:53:22 -0400 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: , , Message-ID: > To: python-ideas at python.org > From: breamoreboy at yahoo.co.uk > Date: Sun, 18 Oct 2015 00:28:01 +0100 > Subject: Re: [Python-ideas] Discussion about a 'xor' keyword? > > On 17/10/2015 23:50, Emile van Sebille wrote: > > Do you have additional arguments that weren't covered in the PEPs > > discussing the addition of an XOR operator? > > > > Emile > > > > The only one I can find that refers specifically to xor is PEP 225 -- > Elementwise/Objectwise Operators[1]. Which others have I missed? > > [1] https://www.python.org/dev/peps/pep-0225/ > > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence That's also what I found, but it very clearly refers to the various operators which are type-overridable, so I think it doesn't really matter here (as I stated in my original post). -Emanuel Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From pyideas at rebertia.com Sun Oct 18 02:40:54 2015 From: pyideas at rebertia.com (Chris Rebert) Date: Sat, 17 Oct 2015 17:40:54 -0700 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: Message-ID: On Sat, Oct 17, 2015 at 3:17 PM, Emanuel Barry wrote: > You've probably read this title with skepticism. I know I would. > > As the title says. I am curious as to engage in the discussion, and see what > everyone's thoughts are on it. I know this has been discussed in the past > several years ago ( > https://mail.python.org/pipermail/python-3000/2007-December/011560.html ) > and on various websites already. I've found myself writing code that would > be simpler with a xor keyword or similar syntax, and this is why I'm > suggesting this. There were a couple times in the past when I found myself wanting a "xor" operator for clarity/aesthetics. In the most recent case, it was to express a validity condition for rows in an SQL DB table which was acting like a tagged union, so only 1 of a subset of the columns in a given row could be non-NULL. That case would have been served just fine by a simple version of xor that always gave a bool result. However, it ultimately turned out that what I really wanted was an "exactly 1 of these N subconditions must be true" operation, which doesn't correspond to conventional definitions of XOR (they instead calculate a parity bit; see http://mathworld.wolfram.com/XOR.html ). So xor would only truly have been useful in cases when N just-so-happens to be 2, and therefore I no longer hunger for a "xor" keyword. I would be interested to hear what your concrete use cases for xor are. Cheers, Chris -- http://chrisrebert.com From pyideas at rebertia.com Sun Oct 18 02:50:36 2015 From: pyideas at rebertia.com (Chris Rebert) Date: Sat, 17 Oct 2015 17:50:36 -0700 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: Message-ID: On Sat, Oct 17, 2015 at 3:17 PM, Emanuel Barry wrote: > You've probably read this title with skepticism. I know I would. > > As the title says. I am curious as to engage in the discussion, and see what > everyone's thoughts are on it. I know this has been discussed in the past > several years ago ( > https://mail.python.org/pipermail/python-3000/2007-December/011560.html ) > and on various websites already. This 2009 discussion was more vigorous: https://mail.python.org/pipermail/python-list/2009-July/543956.html Cheers, Chris From emile at fenx.com Sun Oct 18 03:05:09 2015 From: emile at fenx.com (Emile van Sebille) Date: Sat, 17 Oct 2015 18:05:09 -0700 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: Message-ID: On 10/17/2015 4:28 PM, Mark Lawrence wrote: > On 17/10/2015 23:50, Emile van Sebille wrote: >> Do you have additional arguments that weren't covered in the PEPs >> discussing the addition of an XOR operator? >> >> Emile >> > > The only one I can find that refers specifically to xor is PEP 225 -- > Elementwise/Objectwise Operators[1]. Which others have I missed? > > [1] https://www.python.org/dev/peps/pep-0225/ > It's also touched upon in 207. I thought I saw it mentioned in at least one other, but that may well have been referring to the bitwise operator. Emile From vgr255 at live.ca Sun Oct 18 03:54:54 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Sat, 17 Oct 2015 21:54:54 -0400 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: , Message-ID: > Date: Sat, 17 Oct 2015 17:50:36 -0700 > Subject: Re: [Python-ideas] Discussion about a 'xor' keyword? > From: pyideas at rebertia.com > To: vgr255 at live.ca > CC: python-ideas at python.org > > > This 2009 discussion was more vigorous: > https://mail.python.org/pipermail/python-list/2009-July/543956.html Having a look right now. Looks interesting. > I would be interested to hear what your concrete use cases for xor are. I have a few from the top of my head (not backed by code right now, will post some when I feel like digging into it). For an IRC game bot I work on, we have a special message if someone has two targets available, but a different one if they have only one - although it's the same message, and we made a complicated conditional just for that. A xor keyword would have saved us that trouble. In a library I dug in some time ago (I didn't make it), an exception was raised if two parameters had a True (or False) value - one of them had to be True. I (probably naively) think that some more libraries do that. My friend in college is learning programming, and for his projects he says he really wants to see a 'xor' keyword in, as he has to do some mangling to get what he wants. I'm unaware what his work consists of, however. ------------------ 'xor' would by no means be as common as any existing operator, but I believe it is a good addition to consider. Another upside is that keywords only evaluate the value once, so chaining two wouldn't re-evaluate the value (in the case of a function/property). >From that 2009 discussion, most of the rejections focused on "You can coerce to bool before then perform the xor operations", but I'm considering not coercing to bool here, so I think it is different enough to the other discussion to (at least in my opinion) bring it up one more time. If I wanted boolean coercion, I'd just do it myself. Also, the fact that other people have considered the idea before makes me think that it's not as rare a case as some may think. -Emanuel Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Oct 18 07:18:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 18 Oct 2015 05:18:21 +0000 (UTC) Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: Message-ID: <506401735.2106125.1445145501455.JavaMail.yahoo@mail.yahoo.com> First, it seems like if this operator were to exist, it should be defined in terms of "and", "or", and "not" in one of the usual ways: "a and not b or b and not a", or "(a or b) and not (a and b)". Of course this means it returns False, not None, if both values are falsey or both truthy, but I think that's what you'd want anyway. While you sometimes do want the value of "a and b" or "a or b", it's far more common to just want the truthiness of that value (as in "if a or b:"), and, while None is falsey, it's not the canonical falsey value. But anyway, as my replies below explain, I don't think it should exist. On Saturday, October 17, 2015 6:54 PM, Emanuel Barry wrote: >I have a few from the top of my head (not backed by code right now, will post some when I feel like digging into it). But all of these ideas are just as well served by a function as by a new keyword. There are two reasons for "and" and "or" to be operators. First, functions can't short-circuit. That's obviously not relevant to "xor", because by definition, it has to evaluate both operands. Second, function calls chained in a complicated expression can look ugly compared to operators?compare "or(and(a, b), and(c, d))" to "a and b or c and d". But that isn't relevant to "xor" either. I think most readers would find "a xor b xor c xor d xor e" confusing and have to work through its meaning?and, in fact, they'd find xor(a, xor(b, xor(c, xor(d, e))))" (or the reverse associativity) easier to understand. Worse, I think some readers would _think_ they understand what it means and be wrong (exactly one, or some but not all, or exactly half). And, more importantly, I don't think they'd ever want what it does. It's pretty common to want to check that one or all of three values are true, but when would you ever want to check that an odd number of those three values are true? (You might occasionally want to check that an odd number of the possibly-only-known-at-runtime members of some iterable are true, but for that, you'd use a function akin to "any" and "all", maybe called "parity".) >'xor' would by no means be as common as any existing operator, but I believe it is a good addition to consider. Another upside is that keywords only evaluate the value once, so chaining two wouldn't re-evaluate the value (in the case of a function/property). Again, I don't think you'd want to chain them. But, even if you did, functions also only evaluate their arguments once, so there's no advantage here anyway. For example, try this: def xor(a, b): return a and not b or b and not a xor((print('spam'), 2)[1], (print('eggs'), 3)[1]) Of course "spam" and "eggs" only get printed once, because what gets passed to "xor" isn't the expression "(print('spam'), 2)[1]", it's the value of that expression, "2". And likewise, xor(a, xor(b, c)) isn't going to cause b or c to get evaluated twice. And of course neither would "parity((a, b, c))" or "some((a, b, c))", etc. And I'm not sure what you mean by "property" here. >From that 2009 discussion, most of the rejections focused on "You can coerce to bool before then perform the xor operations", but I'm considering not coercing to bool here, so I think it is different enough to the other discussion to (at least in my opinion) bring it up one more time. If I wanted boolean coercion, I'd just do it myself. But you can also just do it yourself _without_ boolean coercion, so how is this any different? From j.wielicki at sotecware.net Sun Oct 18 13:32:35 2015 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Sun, 18 Oct 2015 13:32:35 +0200 Subject: [Python-ideas] Add appdirs module to stdlib In-Reply-To: References: Message-ID: <56238353.5020807@sotecware.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 01.09.2015 10:00, Philipp A. wrote: > When defining a place for config files, cache files, and so on, > people usually hack around in a OS-dependent, misinformed, and > therefore wrong way. > > Thanks to the tempfile API we at least don?t see people hardcoding > /tmp/ too much. > > There is a beautiful little module that does things right and is > easy to use: appdirs > > TI think this is a *really* good candidate for the stdlib since > this functionality is useful for everything that needs a cache or > config (so not only GUI and CLI applications, but also scripts that > download and cache stuff from the internet for faster re-running) > > probably we should build the API around pathlib, since i found > myself not touching os.path with a barge pole since pathlib > exists. > > i?ll write a PEP about this soon :) Is anyone working on this? I would really like to see the functionality discussed in this thread. I have no deep knowledge about Windows, MacOS or FreeBSD though, which would make it hard for me to take this on my own. If we could get together a team with knowledgeable people from each platform, that would be great. best regards, jwi -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCgAGBQJWI4NSAAoJEMBiAyWXYliKfdwP/Rgi2w2ZyZA2tfFcjZMxh6ox niJYz3Uq2Jcci7VJlvPIObu9ioS1pdc/KdLIqMCO9BZzkm0+V/d9/6TLecBtE73s MDFBkVbBooK/km2xx1HCnfj3vT+Eh8+blPDwNABRKzVdGnuGLfZdQHGvw16NDYPI zKcBboS8Xp1F7Njwdm3ozlW34A5ApLMZV0D2WrtrhZk3PBTbjNVBmKqVdNrBzsyO F04F/8kPLoANBj4tBuI9O1KjnMbfwCN7snbu6LlDwhT8pAGW6kpTzOjxRZZdbjD6 NH6NyLbTjIWgVP8JgQ766vGXAwiatrd0ECQWoYHXgAhuP//KVgfmTBpIqoPPME7Y eGmM8XclzI4FS5ud87bgf9Zgr4xZcycX5J/lcXIp/ZPAajHfyWxVbJPHZDocNw7V ZDlEEfIfkbU4aT5faQutPF95eZgwXOxloOSNZI+u7a8Ke8iaTjeoawtksBlVzYbb 6MOLP6xdVXc+vg/9FFwbiDrLuOQiGv2YYENY2Lj3lMw6jvj6GL31XDKhVcyQQeJn yE1ylKwrBQzq2hw57vLwCJD9Tdn0qgOjA74pgl7SHPvkFjNpPXre5YkMTUxI13jI MGof+oR32WHLaA+Kt0dByzNSJPSdzubc4/zKKAT49G96znXua2MlM1nDdrRmhbHo uihjjtEQzIZ1O/uNMV2n =UtH/ -----END PGP SIGNATURE----- From vgr255 at live.ca Sun Oct 18 14:45:35 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Sun, 18 Oct 2015 08:45:35 -0400 Subject: [Python-ideas] Add appdirs module to stdlib In-Reply-To: <56238353.5020807@sotecware.net> References: , <56238353.5020807@sotecware.net> Message-ID: > To: python-ideas at python.org > From: j.wielicki at sotecware.net > Date: Sun, 18 Oct 2015 13:32:35 +0200 > Subject: Re: [Python-ideas] Add appdirs module to stdlib > > [snip]> > Is anyone working on this? > > I would really like to see the functionality discussed in this thread. > I have no deep knowledge about Windows, MacOS or FreeBSD though, which > would make it hard for me to take this on my own. > > If we could get together a team with knowledgeable people from each > platform, that would be great. > > best regards, > jwi > > [snip] As someone who had to work with temporary files in the past, this appeals to me. I develop on Windows, and the program was not meant to be used on non-Windows platforms, so I used a lot of platform-dependant hacks I wish I could not use. I am by no means a pro, but I would still like to help with this. Cheers,-Emanuel Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Oct 18 17:00:51 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 18 Oct 2015 08:00:51 -0700 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: <506401735.2106125.1445145501455.JavaMail.yahoo@mail.yahoo.com> References: <506401735.2106125.1445145501455.JavaMail.yahoo@mail.yahoo.com> Message-ID: This just isn't going to happen. The combination of the expense of a new keyword, the lack of use cases that can't be done just as easily with a function, and the murky definition make it a non-started. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Sun Oct 18 17:03:43 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Sun, 18 Oct 2015 11:03:43 -0400 Subject: [Python-ideas] Discussion about a 'xor' keyword? In-Reply-To: References: <506401735.2106125.1445145501455.JavaMail.yahoo@mail.yahoo.com>, Message-ID: I figured as much, thanks anyway. Cheers,-Emanuel Barry From: guido at python.org Date: Sun, 18 Oct 2015 08:00:51 -0700 Subject: Re: [Python-ideas] Discussion about a 'xor' keyword? To: abarnert at yahoo.com CC: vgr255 at live.ca; python-ideas at python.org This just isn't going to happen. The combination of the expense of a new keyword, the lack of use cases that can't be done just as easily with a function, and the murky definition make it a non-started. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Oct 20 20:14:31 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 20 Oct 2015 20:14:31 +0200 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> Message-ID: <56268487.9050405@mail.de> On 17.10.2015 04:08, Andrew Barnert via Python-ideas wrote: > Actually, forget all that; it's even simpler. > > At least in recent 3.x, the only thing wrong with inheriting from both > types, assuming you put OrderedDict first, is the __init__ signature. So: > > class OrderedDefaultDict(OrderedDict, defaultdict): > def __init__(self, default_factory=None, *a, **kw): > OrderedDict.__init__(self, *a, **kw) > self.default_factory = default_factory Not saying that inheritance is a bad thing but to me It seems to me that ordering and default values should be orthogonal aspects of the standard dict. Just as Sandi described it here: https://www.youtube.com/watch?v=29MAL8pJImQ Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Oct 20 23:40:16 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Oct 2015 08:40:16 +1100 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <56268487.9050405@mail.de> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> Message-ID: On Wed, Oct 21, 2015 at 5:14 AM, Sven R. Kunze wrote: > Not saying that inheritance is a bad thing but to me It seems to me that > ordering and default values should be orthogonal aspects of the standard > dict. > > Just as Sandi described it here: https://www.youtube.com/watch?v=29MAL8pJImQ Yeah... that video is absolutely correct... for Ruby. In Python, you can use multiple inheritance to do that in the exactly obvious way. class EchoRandomHouse(EchoHouse, RandomHouse): pass When I got to the bit in that video where she says that inheritance paints you into a corner, I went and reimplemented her example code in Python, and it's flawless. It doesn't even matter which order you put the two superclasses, as there's no conflict. ChrisA From carl at oddbird.net Tue Oct 20 23:56:59 2015 From: carl at oddbird.net (Carl Meyer) Date: Tue, 20 Oct 2015 15:56:59 -0600 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> Message-ID: <5626B8AB.5040806@oddbird.net> On 10/20/2015 03:40 PM, Chris Angelico wrote: > On Wed, Oct 21, 2015 at 5:14 AM, Sven R. Kunze wrote: >> Not saying that inheritance is a bad thing but to me It seems to me that >> ordering and default values should be orthogonal aspects of the standard >> dict. >> >> Just as Sandi described it here: https://www.youtube.com/watch?v=29MAL8pJImQ > > Yeah... that video is absolutely correct... for Ruby. In Python, you > can use multiple inheritance to do that in the exactly obvious way. > > class EchoRandomHouse(EchoHouse, RandomHouse): pass > > When I got to the bit in that video where she says that inheritance > paints you into a corner, I went and reimplemented her example code in > Python, and it's flawless. It doesn't even matter which order you put > the two superclasses, as there's no conflict. I think you missed her real point, which applies both to Python and Ruby. In her presentation, it's obscured a bit by too much discussion of "one side or the other" code duplication (which can be "solved" with multiple inheritance). It takes her a couple minutes more to get to the real point, which starts at the slide "inheritance is for specialization, not for sharing code." One symptom of the problem is that using multiple inheritance this way doesn't scale: the number of leaf subclasses you need grows geometrically with the number of behavior knobs. Composition with strategy objects doesn't have this issue. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Wed Oct 21 00:50:21 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Oct 2015 09:50:21 +1100 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <5626B8AB.5040806@oddbird.net> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> Message-ID: On Wed, Oct 21, 2015 at 8:56 AM, Carl Meyer wrote: > I think you missed her real point, which applies both to Python and > Ruby. In her presentation, it's obscured a bit by too much discussion of > "one side or the other" code duplication (which can be "solved" with > multiple inheritance). It takes her a couple minutes more to get to the > real point, which starts at the slide "inheritance is for > specialization, not for sharing code." Sure, but a dictionary with a default handler _is_ a form of specialization - as is a dictionary that preserves order. Both of them behave absolutely identically to a regular dict when you set something, retrieve it, iterate over them, etc, etc, etc. She recommends a massive superclass that's capable of any form of injection; using inheritance allows the dict type to be broadly unaware of the modified dictionaries that exist. Suppose I write a SortedDict that behaves exactly the way any other dictionary does, but when you iterate over its keys, you get them in order. I should be able to do this without tampering with the original dict type. In the composition model, I would need to appeal for an "IterationOrder" feature to be added to the base dict; using inheritance, it's as simple as: class SortedDict(dict): def __iter__(self): yield from sorted(self.keys()) To me, this is the biggest benefit of inheritance: you do NOT have to predict what someone might want to do. I can subclass someone else's object and change how it works. > One symptom of the problem is that using multiple inheritance this way > doesn't scale: the number of leaf subclasses you need grows > geometrically with the number of behavior knobs. Composition with > strategy objects doesn't have this issue. Sure it does... but nobody needs to know about _all_ the leaf subclasses. How many subclasses of object are there in Python? >>> len(object.__subclasses__()) 122 And that in a bare interactive Py3, without importing anything fancy. ('import decimal' raises that to 159, for instance.) Composition has its place, don't get me wrong. But inheritance isn't the attractive nuisance she makes it out to be. It works just fine. ChrisA From chris.barker at noaa.gov Wed Oct 21 01:19:53 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 20 Oct 2015 16:19:53 -0700 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> Message-ID: <2900444672847875025@unknownmsgid> It seems to me that >> ordering and default values should be orthogonal aspects of the standard >> dict. Exactly -- which makes it a perfect candidate for multiple inheritance, I.e. "Mixins" -CHB From steve at pearwood.info Wed Oct 21 03:50:26 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 21 Oct 2015 12:50:26 +1100 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> Message-ID: <20151021015025.GM3813@ando.pearwood.info> On Wed, Oct 21, 2015 at 09:50:21AM +1100, Chris Angelico wrote: > To me, this is the biggest benefit of inheritance: you do NOT have to > predict what someone might want to do. I can subclass someone else's > object and change how it works. I think this is wrong. I think that in general you need to have a fairly good idea of the internal workings of the class before you can inherit from it. Otherwise, how do you know what methods need to be overwritten? Take dict. One nuisance with inheriting from dict is that it isn't sufficient to override __setitem__, you also have to override update and clear as well. And possibly others -- the exact set of which methods depend on which other methods are not documented in the dict API. Given an arbitrary class, how can you possibly tell which methods you need to override, or even which methods are *safe* to override? [...] > Composition has its place, don't get me wrong. But inheritance isn't > the attractive nuisance she makes it out to be. It works just fine. It doesn't work "just fine", it has risks and annoyances, and multiple inheritance even more so. Classes have problems. See Jack Diederich's talk "Stop Writing Classes", or at least stop writing *stupid* classes: http://eev.ee/blog/2013/03/03/the-controller-pattern-is-awful-and-other-oo-heresy/ As far as multiple inheritance, there are real issues with MI that aren't solved by coming up with a nifty solution to the diamond problem. See Michele Simionato's series of essays on super, multiple inheritance, mixins and traits: http://www.artima.com/weblogs/viewpost.jsp?thread=246488 Clearly classes are useful, but they aren't an unalloyed good thing, and inheriting from them has its issues. -- Steve From mehaase at gmail.com Wed Oct 21 04:17:24 2015 From: mehaase at gmail.com (Mark E. Haase) Date: Tue, 20 Oct 2015 22:17:24 -0400 Subject: [Python-ideas] PEP-505: Draft (Coalescing Operators) Message-ID: I've finally finished a reasonably complete draft of PEP-505. It's taken me longer than I expected due to work and also a new addition to our family :) I tried to keep the PEP humorous; but please don't mistake silliness for a lack of seriousness. Let's try to keep separate two different debates: (1) the utility of the proposed operators (2) how to spell them. Arguing about #2 is a waste of time if #1 is a non-starter. If we can achieve some consensus on #1, then with Guido's consent, I think the next logical step would be to conduct a survey of opinions about proposed spellings, just as the authors of PEP-308 did. The survey is a data-gathering exercise, of course, not a democratic vote. The PEP has more details on that. I envision the PEP as being severable: it might be accepted in whole, in part, or not at all. For this reason, there are quite a few alternatives proposed, and some of them are mutually conflicting. The conflicts aren't a problem unless and until any parts of the proposal get the green light. (Again, similar to PEP-308.) https://www.python.org/dev/peps/pep-0505/ -- Mark E. Haase -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed Oct 21 04:58:32 2015 From: random832 at fastmail.com (Random832) Date: Tue, 20 Oct 2015 22:58:32 -0400 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> Message-ID: <87eggporl3.fsf@fastmail.com> Chris Angelico writes: > On Wed, Oct 21, 2015 at 8:56 AM, Carl Meyer wrote: > Sure, but a dictionary with a default handler _is_ a form of > specialization - as is a dictionary that preserves order. Both of them > behave absolutely identically to a regular dict when you set > something, retrieve it, iterate over them, etc, etc, etc. She > recommends a massive superclass that's capable of any form of > injection; using inheritance allows the dict type to be broadly > unaware of the modified dictionaries that exist. Even considering that OrderedDict (or a _proper_ SortedDict - other languages' equivalent class doesn't require hashable keys because it stores items as a sorted list) requires a complete overhaul of how the dictionary stores items? Really, from an implementation standpoint, it seems like a bad idea to even inherit in this case. From guido at python.org Wed Oct 21 05:36:24 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Oct 2015 20:36:24 -0700 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <20151021015025.GM3813@ando.pearwood.info> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <20151021015025.GM3813@ando.pearwood.info> Message-ID: On Tue, Oct 20, 2015 at 6:50 PM, Steven D'Aprano wrote: > On Wed, Oct 21, 2015 at 09:50:21AM +1100, Chris Angelico wrote: > > > To me, this is the biggest benefit of inheritance: you do NOT have to > > predict what someone might want to do. I can subclass someone else's > > object and change how it works. > > I think this is wrong. I think that in general you need to have a fairly > good idea of the internal workings of the class before you can inherit > from it. Otherwise, how do you know what methods need to be overwritten? > Right. Chris's thinking recalls the reason why inheritance at some became popular (I guess it was in the '90s). Steven explains (in the part that I've cut) why many experts have soured on it quite a bit. Personally, I happen to think that inheritance is often useful when a number of classes are designed together (typically all at once and belonging to the same library) and also occasionally when a base class is explicitly and carefully designed to be inherited (there are beautiful things you can do with the template pattern, for example). But inheriting from an implementation that wasn't designed with your use case in mind is often asking for trouble -- if not now, then in the future when a refactored implementation is released. You might interject, that's the fault of the implementation refactoring -- they didn't properly think about interface compatibility. But while it's usually easy enough to keep an interface compatible where it's just the user calling methods on the implementation, the "interface" presented by subclassing is much more complex -- you would have to specify exactly which method's implementation calls which other method, and you'd also have to ensure that the object is in a sane state when it calls that other method, because it *could* be the case that the latter is overridden by a subclass. It's terribly fragile, and better avoided. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Wed Oct 21 07:54:25 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Tue, 20 Oct 2015 22:54:25 -0700 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <20151021015025.GM3813@ando.pearwood.info> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <20151021015025.GM3813@ando.pearwood.info> Message-ID: <56272891.2050500@brenbarn.net> On 2015-10-20 18:50, Steven D'Aprano wrote: > Take dict. One nuisance with inheriting from dict is that it isn't > sufficient to override __setitem__, you also have to override update and > clear as well. And possibly others -- the exact set of which methods > depend on which other methods are not documented in the dict API. Given > an arbitrary class, how can you possibly tell which methods you need to > override, or even which methods are*safe* to override? I've always considered that (and related problems) to be one of Python's warts. And, as you say, it's a documentation flaw. It's true that it's easy to write classes with poorly documented APIs, which makes them hard to extend because you don't know how they work. But I don't think that means subclassing is not a good idea. It means writing classes with clearly-specified APIs is a good idea. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From victor.stinner at gmail.com Wed Oct 21 17:43:57 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Oct 2015 17:43:57 +0200 Subject: [Python-ideas] Add specialized bytecode with guards to functions Message-ID: Hi, I would like to share with you an idea to try to optimize CPython. My previous attempts to optimize CPython failed because they changed the language semantic. For example, it's not possible to replace len('abc') with 3 because it is technically possible to override the builtin len() function. I propose to add a new "FAT" mode to Python to allow to add specialized bytecode to function with guards. Example: >>> import builtins >>> def f(): return len("abc") ... >>> f() 3 >>> def g(): return 3 ... >>> i=f.add(g) # i is the index of the new specialized bytecode >>> f.add_dict_key_guard(i, builtins.__dict__, 'len') >>> f() 3 >>> builtins.len = lambda obj: "len" >>> f() 'len' In this example, the f() function gets a fast version (g()) returning directly 3 instead of calling len("abc"). The optimization is disabled when the builtin len() function is modified (when builtins.__dict__['len'] is modified). (The example is not complete, we need a second guard on the current global namespace, but I wanted to write a short example.) With such machinery, it becomes possible to implement interesting optimizations. Some examples: * inline a function instead of calling it: calling a Python function is "expensive". For example, if you inline the _get_sep() call in posixpath.isabs(), the function becomes 20% faster. * move invariant out of the loop: classic micro-optimization done manually. For example, set "append = obj.append" before the loop and then call "append(item)" instead of "obj.append(item)" in the loop body. On a microbenchmark, it's between 5% (1 item) and 30% faster (10 items or more) * call pure functions at the compilation. A pure function has no side effect. Example: len(str). Technical Challenges ==================== The problem is the cost of "guards". A guard is a check done before calling a specialized function. Example of guards: * Type of a function parameter * Watch a dictionary key * Watch a function (especially it's __code__ attribute) Watching a dictionary key is a very important guard to disable an optimization when a builtin function is modified, when a function is replaced in a namespace, etc. The problem is that a dictionary lookup is not cheap: we have to get the hash value of the key, browse the hash table to find the bucket which may require multiple iterations, then we have to compare the key value, etc. To have faster guard, I propose to create a subtype of dict which provides a global "version" of the dictionary, incremented at each modification (create a new key, modify a key, delete a key) and a "version" per key=>value mapping, incremented each time that the mapping is modified. The lookup can be avoided when the dictionary is not modified. If a different key is modified, we need a lookup, but only once (we store the new global version to avoid a lookup at the next guard check). Limitations =========== Specialized bytecode is build at compilation, *not* at runtime: it's not a just-in-time (JIT) compiler. A JIT can implement even more efficient optimizations and can use better guards. Please continue to use PyPy for best performances! ;-) The name "FAT" comes from the fact that multiple bytecode versions of a function are stored in the memory, so a larger memory footprint and larger .pyc files on disk can be expected. Checking guards can also be more expensive than the optimization of the specialized bytecode. The optimizer should use an heuristic to decide if it's worth to use a specialized bytecode or not depending on the theoric speeup and the cost of guards. My motivation to write FAT Python is that CPython remains the reference implementation where new features are implemented, and other Python implementations still have issues with C extensions. JIT also has some issues, like longer startup time, slow warmup and memory footprint (this one may also be an issue of FAT Python!). Implementation ============== I wrote a proof-of-concept of my idea: https://hg.python.org/sandbox/fatpython/ It's a fork of CPython 3.6. Try it with: hg clone https://hg.python.org/sandbox/fatpython/ cd fatpython ./configure && make ./python -F See bench.py, Lib/posixpath.py (isabs) and Lib/test/test_fat.py for examples of optimizations. The command line -F flag enables the FAT mode. By default, *all* optimizations are disabled, there is a negligible overhead when the FAT mode is not used. In the FAT mode, the dictionary for modules, classes and instances becomes "versionned". Functions gets new methods to support adding specialized bytecode with guards. You can add manually a specialized bytecode for microbenchmarks, but functions are *not* optimized automatically. (See the Roadmap below.) How to write an optimizer? ========================== Currently, my proof-of-oncept is only the machinery to support adding specialized bytecodes with guards. The next step is to generate automatically these specialized versions. IMHO an optimizer should be implemented in Python and not in C, it's easier to write Python code. We can begin with simple optimizations on AST. See for example my astoptimizer which implements basic optimizations: https://bitbucket.org/haypo/astoptimizer/ At the beginning, we may use type hints (PEP 484) and other manual hints to help the optimizer. Later, a profiler learning the most common types of function parameters can be written for automatic optimizations. Example of hints: a list of constants which should not be modified in the application. A global "DEBUG" flag is common in applications, relying on DEBUG=False helps to remove dead code. Roadmap ======= * Finish the proof-of-concept: implement new guards, optimize method calls. For example, guards on module imports are required. Checking if a module is the expected mode can be tricky. * Write an optimizer based on manual hints * Write a profiler to generate hints Right now, my main question is if it will be possible and efficient to optimize classes, not only simple functions, especially classes defined in two different modules. I wrote a proof-of-concept of optimized method (with inlining), but I'm not sure yet that it's safe (don't change Python semantic). For more information on FAT Python, read: https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst So, what do you think? Do you expect real speedup on applications, not only on microbechmarks? Do you know similar optimizations in other scripting languages? Victor Stinner From rosuav at gmail.com Wed Oct 21 17:48:51 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Oct 2015 02:48:51 +1100 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On Thu, Oct 22, 2015 at 2:43 AM, Victor Stinner wrote: > Roadmap > ======= > > * Finish the proof-of-concept: implement new guards, optimize method > calls. For example, guards on module imports are required. Checking > if a module is the expected mode can be tricky. > * Write an optimizer based on manual hints > * Write a profiler to generate hints * Compare against PyPy. For optimized Python execution, my brain immediately jumps to PyPy. How does FAT Python compare in terms of functionality, performance, CPython compatibility, etc? What can it brag as its advantage? ChrisA From ncoghlan at gmail.com Wed Oct 21 18:11:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Oct 2015 18:11:04 +0200 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On 21 October 2015 at 17:48, Chris Angelico wrote: > On Thu, Oct 22, 2015 at 2:43 AM, Victor Stinner > wrote: >> Roadmap >> ======= >> >> * Finish the proof-of-concept: implement new guards, optimize method >> calls. For example, guards on module imports are required. Checking >> if a module is the expected mode can be tricky. >> * Write an optimizer based on manual hints >> * Write a profiler to generate hints > > * Compare against PyPy. > > For optimized Python execution, my brain immediately jumps to PyPy. > How does FAT Python compare in terms of functionality, performance, > CPython compatibility, etc? What can it brag as its advantage? * Full CPython C API compatibility * Reliance on ahead of time compilation means it works for command line scripts, not just long running server processes that allow the PyPy tracing JIT to work its magic As an ahead-of-time optimisation, it's also possible something like this could speed up applications running on PyPy during the JIT warm-up phase. An existing example of this kind of guard-based optimisation exists in the ABC machinery - we globally version the ABC registration graph in order to automatically invalidate caches when a new explicit registration is added. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Wed Oct 21 18:12:56 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Oct 2015 03:12:56 +1100 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On Thu, Oct 22, 2015 at 3:11 AM, Nick Coghlan wrote: >> For optimized Python execution, my brain immediately jumps to PyPy. >> How does FAT Python compare in terms of functionality, performance, >> CPython compatibility, etc? What can it brag as its advantage? > > * Full CPython C API compatibility > * Reliance on ahead of time compilation means it works for command > line scripts, not just long running server processes that allow the > PyPy tracing JIT to work its magic Cool! Thanks for the summary. ChrisA From guido at python.org Wed Oct 21 18:20:02 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Oct 2015 09:20:02 -0700 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: I haven't tried the prototype yet, but I wonder if this mightn't be a useful addition to an existing optimizing Python system, e.g. PyPy or (plug :-) Pyston? On Wed, Oct 21, 2015 at 8:43 AM, Victor Stinner wrote: > Hi, > > I would like to share with you an idea to try to optimize CPython. > > My previous attempts to optimize CPython failed because they changed the > language semantic. For example, it's not possible to replace len('abc') > with 3 because it is technically possible to override the builtin len() > function. > > I propose to add a new "FAT" mode to Python to allow to add specialized > bytecode to function with guards. Example: > > >>> import builtins > >>> def f(): return len("abc") > ... > >>> f() > 3 > > >>> def g(): return 3 > ... > >>> i=f.add(g) # i is the index of the new specialized bytecode > >>> f.add_dict_key_guard(i, builtins.__dict__, 'len') > >>> f() > 3 > > >>> builtins.len = lambda obj: "len" > >>> f() > 'len' > > In this example, the f() function gets a fast version (g()) returning > directly 3 instead of calling len("abc"). The optimization is disabled > when the builtin len() function is modified (when > builtins.__dict__['len'] is modified). (The example is not complete, we > need a second guard on the current global namespace, but I wanted to > write a short example.) > > With such machinery, it becomes possible to implement interesting > optimizations. Some examples: > > * inline a function instead of calling it: calling a Python function is > "expensive". For example, if you inline the _get_sep() call in > posixpath.isabs(), the function becomes 20% faster. > > * move invariant out of the loop: classic micro-optimization done > manually. For example, set "append = obj.append" before the loop and > then call "append(item)" instead of "obj.append(item)" in the loop > body. On a microbenchmark, it's between 5% (1 item) and 30% faster (10 > items or more) > > * call pure functions at the compilation. A pure function has no side > effect. Example: len(str). > > > Technical Challenges > ==================== > > The problem is the cost of "guards". A guard is a check done before > calling a specialized function. Example of guards: > > * Type of a function parameter > * Watch a dictionary key > * Watch a function (especially it's __code__ attribute) > > Watching a dictionary key is a very important guard to disable an > optimization when a builtin function is modified, when a function is > replaced in a namespace, etc. The problem is that a dictionary lookup is > not cheap: we have to get the hash value of the key, browse the hash > table to find the bucket which may require multiple iterations, then we > have to compare the key value, etc. > > To have faster guard, I propose to create a subtype of dict which > provides a global "version" of the dictionary, incremented at each > modification (create a new key, modify a key, delete a key) and a > "version" per key=>value mapping, incremented each time > that the mapping is modified. The lookup can be avoided when the > dictionary is not modified. If a different key is modified, we need a > lookup, but only once (we store the new global version to avoid a lookup > at the next guard check). > > > Limitations > =========== > > Specialized bytecode is build at compilation, *not* at runtime: it's not a > just-in-time (JIT) compiler. A JIT can implement even more efficient > optimizations and can use better guards. Please continue to use PyPy for > best performances! ;-) > > The name "FAT" comes from the fact that multiple bytecode versions of a > function are stored in the memory, so a larger memory footprint > and larger .pyc files on disk can be expected. > > Checking guards can also be more expensive than the optimization of the > specialized bytecode. The optimizer should use an heuristic to decide if > it's worth to use a specialized bytecode or not depending on the theoric > speeup and the cost of guards. > > My motivation to write FAT Python is that CPython remains the reference > implementation where new features are implemented, and other Python > implementations still have issues with C extensions. JIT also has some > issues, like longer startup time, slow warmup and memory footprint (this > one may also be an issue of FAT Python!). > > > Implementation > ============== > > I wrote a proof-of-concept of my idea: > > https://hg.python.org/sandbox/fatpython/ > > It's a fork of CPython 3.6. Try it with: > > hg clone https://hg.python.org/sandbox/fatpython/ > cd fatpython > ./configure && make > ./python -F > > See bench.py, Lib/posixpath.py (isabs) and Lib/test/test_fat.py > for examples of optimizations. > > The command line -F flag enables the FAT mode. By default, *all* > optimizations are disabled, there is a negligible overhead when the > FAT mode is not used. > > In the FAT mode, the dictionary for modules, classes and instances > becomes "versionned". Functions gets new methods to support adding > specialized bytecode with guards. > > You can add manually a specialized bytecode for microbenchmarks, but > functions are *not* optimized automatically. (See the Roadmap below.) > > > How to write an optimizer? > ========================== > > Currently, my proof-of-oncept is only the machinery to support adding > specialized bytecodes with guards. The next step is to generate > automatically these specialized versions. > > IMHO an optimizer should be implemented in Python and not in C, it's > easier to write Python code. We can begin with simple optimizations on > AST. See for example my astoptimizer which implements basic > optimizations: > > https://bitbucket.org/haypo/astoptimizer/ > > At the beginning, we may use type hints (PEP 484) and other manual > hints to help the optimizer. Later, a profiler learning the most common > types of function parameters can be written for automatic optimizations. > Example of hints: a list of constants which should not be modified in the > application. A global "DEBUG" flag is common in applications, relying on > DEBUG=False helps to remove dead code. > > > Roadmap > ======= > > * Finish the proof-of-concept: implement new guards, optimize method > calls. For example, guards on module imports are required. Checking > if a module is the expected mode can be tricky. > * Write an optimizer based on manual hints > * Write a profiler to generate hints > > Right now, my main question is if it will be possible and efficient to > optimize classes, not only simple functions, especially classes defined > in two different modules. I wrote a proof-of-concept of optimized > method (with inlining), but I'm not sure yet that it's safe (don't > change Python semantic). > > For more information on FAT Python, read: > > https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst > > > So, what do you think? Do you expect real speedup on applications, not > only on microbechmarks? Do you know similar optimizations in other > scripting languages? > > Victor Stinner > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Oct 21 18:35:38 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Oct 2015 18:35:38 +0200 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On 21 October 2015 at 17:48, Chris Angelico wrote: > * Compare against PyPy. As written in my email, don't expect better performances than PyPy. PyPy must remain the fatest implementation of Python ;-) The final goal is to see real speedup on non-trivial applications like Django Mercurial, or OpenStack. 2015-10-21 18:11 GMT+02:00 Nick Coghlan : > * Full CPython C API compatibility FYI the FAT mode uses fat.verdict and fat.SpecializedFunction types which are subtypes of dict and types.FunctionTypes, so you keep a full compatibility of the C API. I noticed one issue with the C API: #define PyFunction_Check(op) (Py_TYPE(op) == &PyFunction_Type) This test fails with fat.SpecializedFunction. It means that C extensions must be recompiled to with the FAT mode. I modified the macro to: #define PyFunction_Check(op) \ (Py_TYPE(op) == &PyFunction_Type \ || PyType_IsSubtype(Py_TYPE(op), &PyFunction_Type)) > * Reliance on ahead of time compilation means it works for command > line scripts, not just long running server processes that allow the > PyPy tracing JIT to work its magic Yes, a I wrote in my email, CPython still beats PyPy for fast command line scripts like many Mercurial commands. PyPy JIT takes time to warmup. (Hum, in fact I only read time, I didn't check, sorry.) So yes, I also expect faster performances with static compilation on such use case. > An existing example of this kind of guard-based optimisation exists in > the ABC machinery - we globally version the ABC registration graph in > order to automatically invalidate caches when a new explicit > registration is added. Hum, this is a common cache. CPython has many similar caches, like cache for methods on types, but it's not guard for function calls. I would like to know if someone else tried to write specialized functions with guards in other scripting languages (or even Python?). Victor From victor.stinner at gmail.com Wed Oct 21 18:50:47 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Oct 2015 18:50:47 +0200 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: 2015-10-21 18:20 GMT+02:00 Guido van Rossum : > I haven't tried the prototype yet, but I wonder if this mightn't be a > useful addition to an existing optimizing Python system, e.g. PyPy or (plug > :-) Pyston? I'm not sure that specialized functions with guards would benefit to JIT compilers. A best match is with Cython, pythran and other projects generating a specialized version of a function (in native code), maybe releasing the GIL. It would be cool to be able to use a specialized version under some conditions. For example, if parameters are all floats, use the specialized version optimized by Cython, other fallback to the (slow) regular bytecode. The challenge is to automate everything and make it easy to use. I don't know if Numba would match with this design. Victor From random832 at fastmail.com Wed Oct 21 18:52:01 2015 From: random832 at fastmail.com (Random832) Date: Wed, 21 Oct 2015 12:52:01 -0400 Subject: [Python-ideas] Add specialized bytecode with guards to functions References: Message-ID: <87pp08w4em.fsf@fastmail.com> Victor Stinner writes: > Hi, > > I would like to share with you an idea to try to optimize CPython. > > My previous attempts to optimize CPython failed because they changed the > language semantic. For example, it's not possible to replace len('abc') > with 3 because it is technically possible to override the builtin len() > function. I think this is solving the wrong problem... What about adding an explicit way to "bake in" the values of globals? freeze_globals(func, values) values is a dict with the names of globals and their values, so, you could for example: def foo(): return len('abc') foo = freeze_globals(foo, {'len': len}) # or foo = freeze_globals(foo, __builtins__.__dict__) It would replace any LOAD_GLOBAL for an item that is in the given dictionary (maybe excluding anything that is also stored in the same function) with a LOAD_CONST with a reference to the value from the dictionary. As a second pass, now that the bytecode looks like this: LOAD_CONST ('abc') LOAD_CONST () CALL_FUNCTION 1 This is no longer dependent on anything that can be overridden, and can safely be replaced (given the optimizer knows len is a pure function) with the constant 3. This second pass could also take care of e.g. empty set and frozenset constructors, etc. Decorators could be included to freeze only built-ins, to freeze all globals and built-ins, or to have a list of names to exclude. So you could have @freeze_builtins or @freeze_all_globals for each function that needs heavy optimization. From guido at python.org Wed Oct 21 18:59:50 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Oct 2015 09:59:50 -0700 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On Wed, Oct 21, 2015 at 9:50 AM, Victor Stinner wrote: > 2015-10-21 18:20 GMT+02:00 Guido van Rossum : > > I haven't tried the prototype yet, but I wonder if this mightn't be a > > useful addition to an existing optimizing Python system, e.g. PyPy or > (plug > > :-) Pyston? > > I'm not sure that specialized functions with guards would benefit to > JIT compilers. > Pyston is not primarily a JIT compiler (at least that's my understanding). And my point was the opposite: Pyston (or PyPy, less sure) might benefit from your idea. > A best match is with Cython, pythran and other projects generating a > specialized version of a function (in native code), maybe releasing > the GIL. It would be cool to be able to use a specialized version > under some conditions. For example, if parameters are all floats, use > the specialized version optimized by Cython, other fallback to the > (slow) regular bytecode. The challenge is to automate everything and > make it easy to use. > > I don't know if Numba would match with this design. > Admittedly these are also very attractive-looking pairings! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Oct 21 18:59:54 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Oct 2015 18:59:54 +0200 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: <87pp08w4em.fsf@fastmail.com> References: <87pp08w4em.fsf@fastmail.com> Message-ID: Hi, 2015-10-21 18:52 GMT+02:00 Random832 : > so, you could for example: > > def foo(): > return len('abc') > > foo = freeze_globals(foo, {'len': len}) > # or > foo = freeze_globals(foo, __builtins__.__dict__) You are changing the semantic of Python. In Python, you should be able to override builtin functions. Binding a global name to a local name is a known trick to optimize a function, but you have to write it explicitly, and again you loose the ability to override len. Example: def f(len=len): return len("abc") Here len becomes a fast local variable. > Decorators could be included to freeze only built-ins, to freeze all > globals and built-ins, or to have a list of names to exclude. So you > could have @freeze_builtins or @freeze_all_globals for each function > that needs heavy optimization. I cited "previous attempts" which failed. What you describe is close to my "readonly" attempt which tried to make module and type namespaces readonly: https://hg.python.org/sandbox/readonly/file/tip/READONLY.txt Making namespaces read-only simply doesn't work. It's very common to modify namespaces. I added callbacks to disable optimizations when a namespace is modified. But this design doesn't scale: you add a complexity of O(n) when a namespace is modified, even if optimized functions are never called. My new design (guards) avoids this performance issue. Victor From brett at python.org Wed Oct 21 19:12:34 2015 From: brett at python.org (Brett Cannon) Date: Wed, 21 Oct 2015 17:12:34 +0000 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On Wed, 21 Oct 2015 at 10:00 Guido van Rossum wrote: > On Wed, Oct 21, 2015 at 9:50 AM, Victor Stinner > wrote: > >> 2015-10-21 18:20 GMT+02:00 Guido van Rossum : >> > I haven't tried the prototype yet, but I wonder if this mightn't be a >> > useful addition to an existing optimizing Python system, e.g. PyPy or >> (plug >> > :-) Pyston? >> >> I'm not sure that specialized functions with guards would benefit to >> JIT compilers. >> > > Pyston is not primarily a JIT compiler (at least that's my understanding). > There GitHub project seems to suggest otherwise based on them saying they are leveraging LLVM's JIT: https://github.com/dropbox/pyston#pyston-- . > > And my point was the opposite: Pyston (or PyPy, less sure) might benefit > from your idea. > As would Pyjion (which is really early work and being played with in Dino and my spare time at work): https://github.com/microsoft/pyjion . We would probably love to see at least the dict watcher get into CPython. I know this was something that Unladen Swallow really wanted as well for easy optimizations of built-in functions. Extend dict watchers to a module's or class' __dict__ and you start to get more inlining opportunities as Victor is hoping for. But as a starting point, getting some way to cheaply detect when the built-ins have not been modified from what they are set to by default would be nice to have. -Brett > > >> A best match is with Cython, pythran and other projects generating a >> specialized version of a function (in native code), maybe releasing >> the GIL. It would be cool to be able to use a specialized version >> under some conditions. For example, if parameters are all floats, use >> the specialized version optimized by Cython, other fallback to the >> (slow) regular bytecode. The challenge is to automate everything and >> make it easy to use. >> >> I don't know if Numba would match with this design. >> > > Admittedly these are also very attractive-looking pairings! > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Oct 21 19:23:41 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 21 Oct 2015 19:23:41 +0200 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <20151021015025.GM3813@ando.pearwood.info> Message-ID: <5627CA1D.9040302@mail.de> On 21.10.2015 05:36, Guido van Rossum wrote: > Right. Chris's thinking recalls the reason why inheritance at some > became popular (I guess it was in the '90s). Steven explains (in the > part that I've cut) why many experts have soured on it quite a bit. > Yep. It's because experience shows that the usefulness is limited when it comes to orthogonal aspects. Furthermore, I can remember changing an algorithm's design **because** of the lack of such built-in datastructure. That's always a bad sign. > Personally, I happen to think that inheritance is often useful when a > number of classes are designed together (typically all at once and > belonging to the same library) and also occasionally when a base class > is explicitly and carefully designed to be inherited (there are > beautiful things you can do with the template pattern, for example). > But inheriting from an implementation that wasn't designed with your > use case in mind is often asking for trouble -- if not now, then in > the future when a refactored implementation is released. That's a pretty good observation. > You might interject, that's the fault of the implementation > refactoring -- they didn't properly think about interface > compatibility. But while it's usually easy enough to keep an interface > compatible where it's just the user calling methods on the > implementation, the "interface" presented by subclassing is much more > complex -- you would have to specify exactly which method's > implementation calls which other method, and you'd also have to ensure > that the object is in a sane state when it calls that other method, > because it *could* be the case that the latter is overridden by a > subclass. It's terribly fragile, and better avoided. Maybe, that's one reason why people hesitate to write their own OrderDefaultDict or DefaultOrderedDict. It's just not their responsibility. Best, Sven From p.f.moore at gmail.com Wed Oct 21 19:31:35 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Oct 2015 18:31:35 +0100 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On 21 October 2015 at 18:12, Brett Cannon wrote: > But as a starting point, getting some way to cheaply detect when the > built-ins have not been modified from what they are set to by default would > be nice to have. Would it be worth focusing on mechanisms like this, which would be of general use to various projects, and treating Victor's specific optimisation use case as simply that, a motivating example? At least in the first instance - as Victor pointed out, previous attempts to introduce optimisations have always failed because it's not possible to switch off Python's dynamic nature. So maybe if we had tools which allowed code to more easily *detect* when Python's dynamic features have been used (or rather, when they have *not* been used) we'd be in a better position to develop optimisation strategies that know when they are appropriate? (Another example - Victor mentioned that CPython uses PyFunction_Check, which isn't friendly to function subclasses. A fix to allow function subclasses to work more seamlessly would be another useful general mechanism). Paul From random832 at fastmail.com Wed Oct 21 19:35:32 2015 From: random832 at fastmail.com (Random832) Date: Wed, 21 Oct 2015 13:35:32 -0400 Subject: [Python-ideas] Add specialized bytecode with guards to functions References: <87pp08w4em.fsf@fastmail.com> Message-ID: <878u6ww2e3.fsf@fastmail.com> Victor Stinner writes: > You are changing the semantic of Python. In Python, you should be able > to override builtin functions. What I am proposing is a generalized way to add objects that do not have a literal to a function's constants pool. The global name within the first version of the function is simply used as a placeholder for loading the new constant. It could just as easily be something like def foo(): return _1(abc) foo = add_consts(foo, {'_1':len}) > Binding a global name to a local name is a known trick to optimize a > function, but you have to write it explicitly, and again you loose the > ability to override len. This would also have to be written explicitly, and the whole point is to lose the ability to override things. Explicitly. > Example: > > def f(len=len): > return len("abc") > > Here len becomes a fast local variable. It'd be even faster as a constant, and that way people couldn't call it as f(len=whatever) [so an optimizer doesn't have to deal with the possibility of people doing that] >> Decorators could be included to freeze only built-ins, to freeze all >> globals and built-ins, or to have a list of names to exclude. So you >> could have @freeze_builtins or @freeze_all_globals for each function >> that needs heavy optimization. > > I cited "previous attempts" which failed. > > What you describe is close to my "readonly" attempt which tried to > make module and type namespaces readonly: > https://hg.python.org/sandbox/readonly/file/tip/READONLY.txt This would apply to the function, not the namespace. > Making namespaces read-only simply doesn't work. It's very common to > modify namespaces. I added callbacks to disable optimizations when a > namespace is modified. But this design doesn't scale: you add a > complexity of O(n) when a namespace is modified, even if optimized > functions are never called. Nothing happens when a namespace is modified, because the function is no longer using the namespace in any form. This adds to the cost of defining a function, but in most cases (I'm not 100% sure of the semantics for nested functions, let alone the implementation) this could be done at compile time, so it would be a one-time cost. From srkunze at mail.de Wed Oct 21 19:41:55 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 21 Oct 2015 19:41:55 +0200 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> Message-ID: <5627CE63.9030704@mail.de> On 21.10.2015 00:50, Chris Angelico wrote: > She recommends a massive superclass that's capable of any form of injection Nope. She does not. The "superclass" is not "massive" at all. It is even slimmer as orthogonal aspects are refactored out into separate entities. In fact, it makes it even easier to test and maintain these separate aspects (the core dev should be interested in that). Furthermore, it's, of course, up to debate which aspects should be injectable and which are not. Just imagine, I would need to use several orderings and/or a default value for missing keys: normal_dict = dict() ordered_dict = dict(order=dict.order_by_insert) sorted_dict = dict(order=sorted) sorted_default_dict = dict(order=sorted, default=int) How many subclasses am I supposed to write, maintain and upgrade (in case Guido rewrites his precious dict implementation)? I would even allege that for sufficiently large teams and projects, there are *multiple* implementations with the same intent. Please, no. Best, Sven PS: the instances above are real-world examples, I remember requiring during the course of the last year. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Wed Oct 21 19:57:36 2015 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Wed, 21 Oct 2015 11:57:36 -0600 Subject: [Python-ideas] "else if" as equivalent for "elif" Message-ID: Parse the sequence "else if" equivalently to "elif". This would be 100% backward compatible. It wouldn't require adding any new keywords to the language. It shouldn't be burdensome to maintain, since "elif" only appears in one place in the grammar. And it solves a minor readability complaint. Thoughts? From rymg19 at gmail.com Wed Oct 21 20:05:01 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 21 Oct 2015 13:05:01 -0500 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: Message-ID: The Python Zen says: There should be one-- and preferably only one --obvious way to do it. elif uses less typing, anyway. Why type 3 more characters when you really don't need to? :D On October 21, 2015 12:57:36 PM CDT, Ian Kelly wrote: >Parse the sequence "else if" equivalently to "elif". > >This would be 100% backward compatible. It wouldn't require adding any >new keywords to the language. It shouldn't be burdensome to maintain, >since "elif" only appears in one place in the grammar. And it solves a >minor readability complaint. > >Thoughts? >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Oct 21 20:16:07 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 21 Oct 2015 20:16:07 +0200 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: Message-ID: <5627D667.8020301@mail.de> On 21.10.2015 20:05, Ryan Gonzalez wrote: > The Python Zen says: > > There should be one-- and preferably only one --obvious way to do it. > > elif uses less typing, anyway. Why type 3 more characters when you > really don't need to? :D Because it's one of those "Python looks strange"? ;-) Best, Sven From jonathan at slenders.be Wed Oct 21 20:25:58 2015 From: jonathan at slenders.be (Jonathan Slenders) Date: Wed, 21 Oct 2015 20:25:58 +0200 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <5627CE63.9030704@mail.de> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <5627CE63.9030704@mail.de> Message-ID: Just want to say that I'm happy to see that lately the disadvantages of inheritance (which are already known for a very long time) are getting more attention. It's not bad by definition, but there's so much truth in Sandy her talk and I think for many Python projects, we went way too far into "abusing" inheritance. Actually, it's a bit unfortunate that we made inheritance so user friendly and powerful in Python that for many people it became the logical way to extend or reuse some code. Jonathan 2015-10-21 19:41 GMT+02:00 Sven R. Kunze : > On 21.10.2015 00:50, Chris Angelico wrote: > > She recommends a massive superclass that's capable of any form of injection > > > Nope. She does not. > > The "superclass" is not "massive" at all. It is even slimmer as orthogonal > aspects are refactored out into separate entities. In fact, it makes it > even easier to test and maintain these separate aspects (the core dev > should be interested in that). Furthermore, it's, of course, up to debate > which aspects should be injectable and which are not. > > > Just imagine, I would need to use several orderings and/or a default value > for missing keys: > > normal_dict = dict() > ordered_dict = dict(order=dict.order_by_insert) > sorted_dict = dict(order=sorted) > sorted_default_dict = dict(order=sorted, default=int) > > > How many subclasses am I supposed to write, maintain and upgrade (in case > Guido rewrites his precious dict implementation)? I would even allege that > for sufficiently large teams and projects, there are *multiple* > implementations with the same intent. > > > Please, no. > > > Best, > Sven > > > PS: the instances above are real-world examples, I remember requiring > during the course of the last year. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Oct 21 20:43:07 2015 From: brett at python.org (Brett Cannon) Date: Wed, 21 Oct 2015 18:43:07 +0000 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On Wed, 21 Oct 2015 at 10:31 Paul Moore wrote: > On 21 October 2015 at 18:12, Brett Cannon wrote: > > But as a starting point, getting some way to cheaply detect when the > > built-ins have not been modified from what they are set to by default > would > > be nice to have. > > Would it be worth focusing on mechanisms like this, which would be of > general use to various projects, and treating Victor's specific > optimisation use case as simply that, a motivating example? I think so. > At least > in the first instance - as Victor pointed out, previous attempts to > introduce optimisations have always failed because it's not possible > to switch off Python's dynamic nature. So maybe if we had tools which > allowed code to more easily *detect* when Python's dynamic features > have been used (or rather, when they have *not* been used) we'd be in > a better position to develop optimisation strategies that know when > they are appropriate? > Exactly, and especially if the check is cheap. After that you start to potentially get into granularity, e.g. do you care if *any* built-in changed or do you only care if a *specific* built-in changed, and how do either approach affect performance? The trick is not breaking compatibility and not hurting performance in the general case. But yes, knowing when the uncommon case of overriding a built-in occurs would allow for a bunch of optimization techniques from inlining to using a faster implementation that is compatible with the standard built-in. -Brett > > (Another example - Victor mentioned that CPython uses > PyFunction_Check, which isn't friendly to function subclasses. A fix > to allow function subclasses to work more seamlessly would be another > useful general mechanism). > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Oct 21 21:10:03 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 21 Oct 2015 12:10:03 -0700 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <5627D667.8020301@mail.de> References: <5627D667.8020301@mail.de> Message-ID: On Oct 21, 2015, at 11:16, Sven R. Kunze wrote: > >> On 21.10.2015 20:05, Ryan Gonzalez wrote: >> The Python Zen says: >> >> There should be one-- and preferably only one --obvious way to do it. >> >> elif uses less typing, anyway. Why type 3 more characters when you really don't need to? :D > > Because it's one of those "Python looks strange"? ;-) Unless you also deprecate "elif", Python will still look strange?it'll just look inconsistently strange, which is even worse than consistently strange. Besides, it's not as if "else if" reads like English any more than "elif" does. In English, you'd use a semicolon and an "if" for all but the last, and "or if" for the last (or, if there's only one, maybe "otherwise if" or "but if", depending on the connotations). Also, the number of people coming to Python from C and its descendants (especially Java and JavaScript) is high enough that the false impression given by "else if" (that it's just an if statement controlled by an else clause) could be harmful. Yes, once you think about it, the fact that there's no colon after the "else" and only one level of indentation instead of two makes sense, but you shouldn't have to think about it. From random832 at fastmail.com Wed Oct 21 21:21:18 2015 From: random832 at fastmail.com (Random832) Date: Wed, 21 Oct 2015 15:21:18 -0400 Subject: [Python-ideas] "else if" as equivalent for "elif" References: <5627D667.8020301@mail.de> Message-ID: <87twpkuixd.fsf@fastmail.com> Andrew Barnert via Python-ideas writes: > Also, the number of people coming to Python from C and its descendants > (especially Java and JavaScript) is high enough that the false > impression given by "else if" (that it's just an if statement > controlled by an else clause) could be harmful. Yes, once you think > about it, the fact that there's no colon after the "else" and only one > level of indentation instead of two makes sense, but you shouldn't > have to think about it. Could go all the way and actually allow "else: if ...:" or any other combination such as "for ...: if...:" with a single level of indentation. I bet that's been proposed and rejected before. From python at mrabarnett.plus.com Wed Oct 21 21:26:47 2015 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 21 Oct 2015 20:26:47 +0100 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <5627D667.8020301@mail.de> References: <5627D667.8020301@mail.de> Message-ID: <5627E6F7.50200@mrabarnett.plus.com> On 2015-10-21 19:16, Sven R. Kunze wrote: > On 21.10.2015 20:05, Ryan Gonzalez wrote: >> The Python Zen says: >> >> There should be one-- and preferably only one --obvious way to do it. >> >> elif uses less typing, anyway. Why type 3 more characters when you >> really don't need to? :D > > Because it's one of those "Python looks strange"? ;-) > The C preprocessor has #elif. Is that strange too? :-) (It could be argued that it _is_ strange that the C preprocessor has an explicitly-terminated "#if ... #endif", whereas the C language itself doesn't have an explicitly-terminated "if".) From rymg19 at gmail.com Wed Oct 21 21:29:32 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 21 Oct 2015 14:29:32 -0500 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <5627E6F7.50200@mrabarnett.plus.com> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: On October 21, 2015 2:26:47 PM CDT, MRAB wrote: >On 2015-10-21 19:16, Sven R. Kunze wrote: >> On 21.10.2015 20:05, Ryan Gonzalez wrote: >>> The Python Zen says: >>> >>> There should be one-- and preferably only one --obvious way to do >it. >>> >>> elif uses less typing, anyway. Why type 3 more characters when you >>> really don't need to? :D >> >> Because it's one of those "Python looks strange"? ;-) >> >The C preprocessor has #elif. Is that strange too? :-) > >(It could be argued that it _is_ strange that the C preprocessor has an >explicitly-terminated "#if ... #endif", whereas the C language itself >doesn't have an explicitly-terminated "if".) > It's the C preprocessor. How is it NOT strange? :) >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From abarnert at yahoo.com Wed Oct 21 21:27:29 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 21 Oct 2015 12:27:29 -0700 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <5627CE63.9030704@mail.de> Message-ID: <3840798D-28DD-4AF3-A6A2-4BAA1D89442A@yahoo.com> On Oct 21, 2015, at 11:25, Jonathan Slenders wrote: > > Just want to say that I'm happy to see that lately the disadvantages of inheritance (which are already known for a very long time) are getting more attention. > It's not bad by definition, but there's so much truth in Sandy her talk and I think for many Python projects, we went way too far into "abusing" inheritance. > Actually, it's a bit unfortunate that we made inheritance so user friendly and powerful in Python that for many people it became the logical way to extend or reuse some code. One of the most important things people have learned about OO over the past two decades is that subtyping, implementation extension, and implementation mixins, and interface-extending mixins (think collections.abc.Sequence adding count for you) are all different things. Compared to the other languages in existence at the time of Python 1.x or even the 2.2/2.3 changeover, it's hard to fault Python. The fact that it can easily be used to write bad code is a little unfortunate, but the fact that it can also easily be used to write good code, when other languages either don't allow some things to be expressed, force them to be expressed in clumsy or limited ways, or force you to misuse inappropriate features instead more than makes up for it. In particular, any language that has fixed structure layouts and vtables makes is much more unfriendly to both kinds of mixins, and makes interface subtyping clumsy; Python has no such problems. Yeah, maybe we could design something better in 2015, but based on the knowledge people had at the time? From abarnert at yahoo.com Wed Oct 21 21:45:50 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 21 Oct 2015 12:45:50 -0700 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: <878u6ww2e3.fsf@fastmail.com> References: <87pp08w4em.fsf@fastmail.com> <878u6ww2e3.fsf@fastmail.com> Message-ID: <1C6ACCBD-EB5F-49EB-85E2-1FE2B1003CBC@yahoo.com> On Oct 21, 2015, at 10:35, Random832 wrote: > > Victor Stinner > writes: >> You are changing the semantic of Python. In Python, you should be able >> to override builtin functions. > > What I am proposing is a generalized way to add objects that do not have > a literal to a function's constants pool. The global name within the > first version of the function is simply used as a placeholder for > loading the new constant. > > It could just as easily be something like > > def foo(): > return _1(abc) > > foo = add_consts(foo, {'_1':len}) > >> Binding a global name to a local name is a known trick to optimize a >> function, but you have to write it explicitly, and again you loose the >> ability to override len. > > This would also have to be written explicitly, and the whole point is to > lose the ability to override things. Explicitly. > >> Example: >> >> def f(len=len): >> return len("abc") >> >> Here len becomes a fast local variable. > > It'd be even faster as a constant, and that way people couldn't call it > as f(len=whatever) [so an optimizer doesn't have to deal with the > possibility of people doing that] > >>> Decorators could be included to freeze only built-ins, to freeze all >>> globals and built-ins, or to have a list of names to exclude. So you >>> could have @freeze_builtins or @freeze_all_globals for each function >>> that needs heavy optimization. >> >> I cited "previous attempts" which failed. >> >> What you describe is close to my "readonly" attempt which tried to >> make module and type namespaces readonly: >> https://hg.python.org/sandbox/readonly/file/tip/READONLY.txt > > This would apply to the function, not the namespace. > >> Making namespaces read-only simply doesn't work. It's very common to >> modify namespaces. I added callbacks to disable optimizations when a >> namespace is modified. But this design doesn't scale: you add a >> complexity of O(n) when a namespace is modified, even if optimized >> functions are never called. > > Nothing happens when a namespace is modified, because the function is no > longer using the namespace in any form. This adds to the cost of > defining a function, but in most cases (I'm not 100% sure of the > semantics for nested functions, let alone the implementation) this could > be done at compile time, so it would be a one-time cost. Nested functions are just like top-level functions: at compile time, the function body is compiled to a code object (bytecode, a list of free variables, etc.) that gets stored as a hidden constant; at runtime, all that happens is building a function object out of it (and, if def rather than lambda, binding it to a name). The only way nested functions are more costly at runtime is that they're more likely to have free variables that have to get bound to cells, but even that isn't a big deal. Anyway, if you just assume that free variables can never be optimized out (which seems fair for a first implementation), you should be able to treat nested functions the same as global functions. If you want to later optimize out free variables bound to constants, you need to add some way to guard the local namespace?which isn't really a dict, but an array in each frame (although you only have to guard the frames that have cells bound to some closure). That seems doable too, but probably worth skipping in the first version. From geoffspear at gmail.com Wed Oct 21 21:49:36 2015 From: geoffspear at gmail.com (Geoffrey Spear) Date: Wed, 21 Oct 2015 15:49:36 -0400 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <5627E6F7.50200@mrabarnett.plus.com> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: On Wed, Oct 21, 2015 at 3:26 PM, MRAB wrote: > > The C preprocessor has #elif. Is that strange too? :-) > > (It could be argued that it _is_ strange that the C preprocessor has an > explicitly-terminated "#if ... #endif", whereas the C language itself > doesn't have an explicitly-terminated "if".) > > "We stole this short spelling from C" shouldn't be an argument in favor of any spelling, in my opinion. (Not creating a second way to do something or deprecating a language keyword is a much stronger argument, of course; I'm as opposed to this proposal as I'd be to a propose to rename strptime and strftime, but I'd also think "but that's how C spells them!" would be a bad argument in that case. Don't even get me started about Popen objects...) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed Oct 21 22:00:41 2015 From: random832 at fastmail.com (Random832) Date: Wed, 21 Oct 2015 16:00:41 -0400 Subject: [Python-ideas] Add specialized bytecode with guards to functions References: <87pp08w4em.fsf@fastmail.com> <878u6ww2e3.fsf@fastmail.com> <1C6ACCBD-EB5F-49EB-85E2-1FE2B1003CBC@yahoo.com> Message-ID: <87pp08htzq.fsf@fastmail.com> Andrew Barnert via Python-ideas writes: > Nested functions are just like top-level functions: The point is that my suggested feature is a decorator (or decorator-like function) attached to the function, to modify the function to replace free variables with constants. This means if the decorator is attached to the inner function it gets executed every time the outer function is executed, and if it is attached to the outer function we have to decide whether it will walk through that function's inner functions and change them too. From abarnert at yahoo.com Wed Oct 21 22:01:04 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 21 Oct 2015 13:01:04 -0700 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <5627E6F7.50200@mrabarnett.plus.com> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: <1E164613-20A4-4616-B392-3CB19957D74B@yahoo.com> On Oct 21, 2015, at 12:26, MRAB wrote: > >> On 2015-10-21 19:16, Sven R. Kunze wrote: >>> On 21.10.2015 20:05, Ryan Gonzalez wrote: >>> The Python Zen says: >>> >>> There should be one-- and preferably only one --obvious way to do it. >>> >>> elif uses less typing, anyway. Why type 3 more characters when you >>> really don't need to? :D >> >> Because it's one of those "Python looks strange"? ;-) > The C preprocessor has #elif. Is that strange too? :-) > > (It could be argued that it _is_ strange that the C preprocessor has an > explicitly-terminated "#if ... #endif", whereas the C language itself > doesn't have an explicitly-terminated "if".) C++ of course solved that by embedding a whole compile-time language that's even less like C than the preprocessor, where the clauses have to be bounded in angle brackets, removing all ambiguity and leading to nice readable things like this: typedef typename if_c::type> ::type type; Compare to: if (spam) type = SpamType; else if (eggs) type = EggsType; else type = CheeseType; The second one leaves it unclear to the reader that the final else is actually part of the second if which is itself controlled by the first if's else. If you indebted this "properly" it would be a lot uglier. Which is the whole problem elif is meant to solve. But the first version makes the tree structure the clearest thing in the whole construction, without needing elif, so it's clearly better. ;) From alexander.belopolsky at gmail.com Wed Oct 21 22:09:06 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 21 Oct 2015 16:09:06 -0400 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: On Wed, Oct 21, 2015 at 3:49 PM, Geoffrey Spear wrote: > "We stole this short spelling from C" shouldn't be an argument in favor of > any spelling, in my opinion. Can you explain your opinion? C is by far the most influential language in the history of computing. As a source of keywords it is second only to English. -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Oct 21 22:24:04 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 21 Oct 2015 22:24:04 +0200 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: <5627F464.5030700@mail.de> On 21.10.2015 21:29, Ryan Gonzalez wrote: > O October 21, 2015 2:26:47 PM CDT, MRAB wrote: >> On 2015-10-21 19:16, Sven R. Kunze wrote: >>> Because it's one of those "Python looks strange"? ;-) >> The C preprocessor has #elif. Is that strange too? :-) >> >> (It could be argued that it _is_ strange that the C preprocessor has an >> explicitly-terminated "#if ... #endif", whereas the C language itself >> doesn't have an explicitly-terminated "if".) >> > It's the C preprocessor. How is it NOT strange? :) :D You made my day! Best, Sven From python at mrabarnett.plus.com Wed Oct 21 22:30:53 2015 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 21 Oct 2015 21:30:53 +0100 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: <5627F5FD.8080707@mrabarnett.plus.com> On 2015-10-21 20:49, Geoffrey Spear wrote: > > > On Wed, Oct 21, 2015 at 3:26 PM, MRAB > wrote: > > The C preprocessor has #elif. Is that strange too? :-) > > (It could be argued that it _is_ strange that the C preprocessor has an > explicitly-terminated "#if ... #endif", whereas the C language itself > doesn't have an explicitly-terminated "if".) > > > "We stole this short spelling from C" shouldn't be an argument in favor > of any spelling, in my opinion. > > (Not creating a second way to do something or deprecating a language > keyword is a much stronger argument, of course; I'm as opposed to this > proposal as I'd be to a propose to rename strptime and strftime, but I'd > also think "but that's how C spells them!" would be a bad argument in > that case. Don't even get me started about Popen objects...) > Modula-2 spells it "elsif". The problem there is that you then have to be very careful when talking about it to someone else: "Did you say 'elsif' or 'else if'?" From kmod at dropbox.com Wed Oct 21 22:55:27 2015 From: kmod at dropbox.com (Kevin Modzelewski) Date: Wed, 21 Oct 2015 13:55:27 -0700 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: I don't think there will be that many cases that len() gets called on a constant object, especially in performance-sensitive code since that would be easy to spot and get rid of. And without some good knowledge of the argument, I would guess that it's only marginally helpful to know that "len" is the builtin len. I'm not saying that CPython couldn't add techniques like this, but I think it might need to go decently far into JIT territory to really make use of it. For example, which functions would have multiple versions generated? There could be startup+memory overheads if it gets applied to all functions, so maybe there needs to be some sort of light profiling to determine when to produce the optimized version. I'm biased but I think this should be left to the JITs :) On the other hand, I think it would be pretty interesting for the core python community to come up with source-level constructs that could help with this sort of thing. For example, one idea is to make it canonical to do something like: def f(): from __builtin__ import len return len("abc") and then let implementations pattern-match this to statically resolve the import. This doesn't quite work since you can mutate __builtin__ (or override __import__); maybe a new module name could be used, or maybe there could be some sort of "bind_globals()" decorator. I think these are interesting since they could improve name-resolution in a targeted way, and also give more information to the JITs. kmod On Wed, Oct 21, 2015 at 8:43 AM, Victor Stinner wrote: > Hi, > > I would like to share with you an idea to try to optimize CPython. > > My previous attempts to optimize CPython failed because they changed the > language semantic. For example, it's not possible to replace len('abc') > with 3 because it is technically possible to override the builtin len() > function. > > I propose to add a new "FAT" mode to Python to allow to add specialized > bytecode to function with guards. Example: > > >>> import builtins > >>> def f(): return len("abc") > ... > >>> f() > 3 > > >>> def g(): return 3 > ... > >>> i=f.add(g) # i is the index of the new specialized bytecode > >>> f.add_dict_key_guard(i, builtins.__dict__, 'len') > >>> f() > 3 > > >>> builtins.len = lambda obj: "len" > >>> f() > 'len' > > In this example, the f() function gets a fast version (g()) returning > directly 3 instead of calling len("abc"). The optimization is disabled > when the builtin len() function is modified (when > builtins.__dict__['len'] is modified). (The example is not complete, we > need a second guard on the current global namespace, but I wanted to > write a short example.) > > With such machinery, it becomes possible to implement interesting > optimizations. Some examples: > > * inline a function instead of calling it: calling a Python function is > "expensive". For example, if you inline the _get_sep() call in > posixpath.isabs(), the function becomes 20% faster. > > * move invariant out of the loop: classic micro-optimization done > manually. For example, set "append = obj.append" before the loop and > then call "append(item)" instead of "obj.append(item)" in the loop > body. On a microbenchmark, it's between 5% (1 item) and 30% faster (10 > items or more) > > * call pure functions at the compilation. A pure function has no side > effect. Example: len(str). > > > Technical Challenges > ==================== > > The problem is the cost of "guards". A guard is a check done before > calling a specialized function. Example of guards: > > * Type of a function parameter > * Watch a dictionary key > * Watch a function (especially it's __code__ attribute) > > Watching a dictionary key is a very important guard to disable an > optimization when a builtin function is modified, when a function is > replaced in a namespace, etc. The problem is that a dictionary lookup is > not cheap: we have to get the hash value of the key, browse the hash > table to find the bucket which may require multiple iterations, then we > have to compare the key value, etc. > > To have faster guard, I propose to create a subtype of dict which > provides a global "version" of the dictionary, incremented at each > modification (create a new key, modify a key, delete a key) and a > "version" per key=>value mapping, incremented each time > that the mapping is modified. The lookup can be avoided when the > dictionary is not modified. If a different key is modified, we need a > lookup, but only once (we store the new global version to avoid a lookup > at the next guard check). > > > Limitations > =========== > > Specialized bytecode is build at compilation, *not* at runtime: it's not a > just-in-time (JIT) compiler. A JIT can implement even more efficient > optimizations and can use better guards. Please continue to use PyPy for > best performances! ;-) > > The name "FAT" comes from the fact that multiple bytecode versions of a > function are stored in the memory, so a larger memory footprint > and larger .pyc files on disk can be expected. > > Checking guards can also be more expensive than the optimization of the > specialized bytecode. The optimizer should use an heuristic to decide if > it's worth to use a specialized bytecode or not depending on the theoric > speeup and the cost of guards. > > My motivation to write FAT Python is that CPython remains the reference > implementation where new features are implemented, and other Python > implementations still have issues with C extensions. JIT also has some > issues, like longer startup time, slow warmup and memory footprint (this > one may also be an issue of FAT Python!). > > > Implementation > ============== > > I wrote a proof-of-concept of my idea: > > https://hg.python.org/sandbox/fatpython/ > > It's a fork of CPython 3.6. Try it with: > > hg clone https://hg.python.org/sandbox/fatpython/ > cd fatpython > ./configure && make > ./python -F > > See bench.py, Lib/posixpath.py (isabs) and Lib/test/test_fat.py > for examples of optimizations. > > The command line -F flag enables the FAT mode. By default, *all* > optimizations are disabled, there is a negligible overhead when the > FAT mode is not used. > > In the FAT mode, the dictionary for modules, classes and instances > becomes "versionned". Functions gets new methods to support adding > specialized bytecode with guards. > > You can add manually a specialized bytecode for microbenchmarks, but > functions are *not* optimized automatically. (See the Roadmap below.) > > > How to write an optimizer? > ========================== > > Currently, my proof-of-oncept is only the machinery to support adding > specialized bytecodes with guards. The next step is to generate > automatically these specialized versions. > > IMHO an optimizer should be implemented in Python and not in C, it's > easier to write Python code. We can begin with simple optimizations on > AST. See for example my astoptimizer which implements basic > optimizations: > > https://bitbucket.org/haypo/astoptimizer/ > > At the beginning, we may use type hints (PEP 484) and other manual > hints to help the optimizer. Later, a profiler learning the most common > types of function parameters can be written for automatic optimizations. > Example of hints: a list of constants which should not be modified in the > application. A global "DEBUG" flag is common in applications, relying on > DEBUG=False helps to remove dead code. > > > Roadmap > ======= > > * Finish the proof-of-concept: implement new guards, optimize method > calls. For example, guards on module imports are required. Checking > if a module is the expected mode can be tricky. > * Write an optimizer based on manual hints > * Write a profiler to generate hints > > Right now, my main question is if it will be possible and efficient to > optimize classes, not only simple functions, especially classes defined > in two different modules. I wrote a proof-of-concept of optimized > method (with inlining), but I'm not sure yet that it's safe (don't > change Python semantic). > > For more information on FAT Python, read: > > https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst > > > So, what do you think? Do you expect real speedup on applications, not > only on microbechmarks? Do you know similar optimizations in other > scripting languages? > > Victor Stinner > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Oct 21 22:56:41 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 21 Oct 2015 22:56:41 +0200 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <3840798D-28DD-4AF3-A6A2-4BAA1D89442A@yahoo.com> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <5627CE63.9030704@mail.de> <3840798D-28DD-4AF3-A6A2-4BAA1D89442A@yahoo.com> Message-ID: <5627FC09.7060106@mail.de> On 21.10.2015 21:27, Andrew Barnert wrote: > On Oct 21, 2015, at 11:25, Jonathan Slenders wrote: >> Just want to say that I'm happy to see that lately the disadvantages of inheritance (which are already known for a very long time) are getting more attention. >> It's not bad by definition, but there's so much truth in Sandy her talk and I think for many Python projects, we went way too far into "abusing" inheritance. >> Actually, it's a bit unfortunate that we made inheritance so user friendly and powerful in Python that for many people it became the logical way to extend or reuse some code. > One of the most important things people have learned about OO over the past two decades is that subtyping, implementation extension, and implementation mixins, and interface-extending mixins (think collections.abc.Sequence adding count for you) are all different things. Compared to the other languages in existence at the time of Python 1.x or even the 2.2/2.3 changeover, it's hard to fault Python. The fact that it can easily be used to write bad code is a little unfortunate, but the fact that it can also easily be used to write good code, when other languages either don't allow some things to be expressed, force them to be expressed in clumsy or limited ways, or force you to misuse inappropriate features instead more than makes up for it. In particular, any language that has fixed structure layouts and vtables makes is much more unfriendly to both kinds of mixins, and makes interface subtyping clumsy; Python has no such problems. Yeah, maybe we could design something better in 2015, but based on the knowledge people had at the time? True. Just because everything is possible does not necessarily mean one should do it. @Jonathan Inheritance has its disadvantages. The bigger the class hierarchy, the uglier it becomes. (We have this with the django view class system.) I see the same coming for the dict discussion if people insist on having separate leaf classes for all possible orthogonal aspects. That does not scale and it's hard to test ( --> @core-devs ). Something that's become clearer and clearer to me is that *nobody could have known this in advance*. It seemed like a good idea to start with several different base classes for each aspect (not knowing that they could be combine later). But now, people start to realize they might need these different aspects in a single implementation. So, it's the time to think about how to solve it in a scalable and maintainable way. Multiple inheritance might be a solution, composition is another. This time I think composition is the better one. Best, Sven PS: Funny coincidence, Sandi uses the aspect "order" for the house. The same as a dict could have an aspect "order". -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Wed Oct 21 23:03:00 2015 From: antoine at python.org (Antoine Pitrou) Date: Wed, 21 Oct 2015 21:03:00 +0000 (UTC) Subject: [Python-ideas] Add specialized bytecode with guards to functions References: Message-ID: Brett Cannon writes: > > Exactly, and especially if the check is cheap. After that you start to potentially get into granularity, e.g. do you care if any?built-in changed or do you only care if a specific?built-in changed, and how do either approach affect performance? The trick is not breaking compatibility and not hurting performance in the general case. But yes, knowing when the uncommon case of overriding a built-in occurs would allow for a bunch of optimization techniques from inlining to using a faster implementation that is compatible with the standard built-in. I think adding a version number to dicts would be a very cheap change and would potentially help several CPython-based projects (including perhaps Numba some day, though nowadays it wouldn't benefit). However, I would also like to challenge the idea that an accelerator has to be 100% compatible. If your API is opt-in (which Numba's is), then you can happily relax some constraints (provided it is documented) in exchange for much faster code. Our experience with Numba is that nobody cares if you can't monkeypatch the len() builtin, for example. (Numba actually goes quite far in that direction since e.g. its integers are fixed-width; the speed of generated code, for supported Python constructs, usually approaches or reaches C, Fortran or even hand-tuned assembly speed) Regards Antoine. From antoine at python.org Wed Oct 21 23:09:07 2015 From: antoine at python.org (Antoine Pitrou) Date: Wed, 21 Oct 2015 21:09:07 +0000 (UTC) Subject: [Python-ideas] Add specialized bytecode with guards to functions References: Message-ID: Victor Stinner writes: > > 2015-10-21 18:20 GMT+02:00 Guido van Rossum : > > I haven't tried the prototype yet, but I wonder if this mightn't be a > > useful addition to an existing optimizing Python system, e.g. PyPy or (plug > > Pyston? > > I'm not sure that specialized functions with guards would benefit to > JIT compilers. > > A best match is with Cython, pythran and other projects generating a > specialized version of a function (in native code), maybe releasing > the GIL. It would be cool to be able to use a specialized version > under some conditions. For example, if parameters are all floats, use > the specialized version optimized by Cython, other fallback to the > (slow) regular bytecode. The challenge is to automate everything and > make it easy to use. > > I don't know if Numba would match with this design. Numba is primarily a JIT compiler but next release will also feature a limited AOT compilation mode (meaning you can compile extension modules for given explicit signatures of whatever functions you want to compile, and don't need Numba to import or run those functions). Documentation preview here: http://numba.pydata.org/numba-doc/dev/user/pycc.html As a JIT compiler, however, Numba is special as it's opt-in and doesn't claim to support the whole range of Python semantics (far from it). Numba takes your Python code and pretty much assumes it has the same dynamicity as Fortran and C++ code, runs type inference and compiles a very fast version out of it, for the CPU (by default) or for the GPU (with a specific programming model). Regards Antoine. From victor.stinner at gmail.com Wed Oct 21 23:11:01 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Oct 2015 23:11:01 +0200 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: Hi, 2015-10-21 22:55 GMT+02:00 Kevin Modzelewski : > I don't think there will be that many cases that len() gets called on a > constant object, especially in performance-sensitive code since that would > be easy to spot and get rid of. Hum, I guess that almost no code calls len() on a constant string :-) In my experience astoptimizer, I noticed that it becomes more common when you combine it with constant folding optimization. Many strings are "constants" in Python and commonly tested. Example: "if sys.platform.startswith('freebsd'): ...". If sys.platform is "linux", it's dead code and the whole if can be removed. > And without some good knowledge of the > argument, I would guess that it's only marginally helpful to know that "len" > is the builtin len. I took the len("abc") example just because it's very easy to understand :-) It's just to explain the principle. Later, if you combine multiple optimizations, I bet that it will become really interesting on real applications. But first we need a framework to make these optimizations legit in Python (don't change Python semantic). The goal is to optimize classes, not only builtin functions. > I'm not saying that CPython couldn't add techniques > like this, but I think it might need to go decently far into JIT territory > to really make use of it. For example, which functions would have multiple > versions generated? There could be startup+memory overheads if it gets > applied to all functions, so maybe there needs to be some sort of light > profiling to determine when to produce the optimized version. I'm biased > but I think this should be left to the JITs :) We will need heuristic to estimate the theoric performance speedup with specialized bytecode, to then decide if it's worth to keep it or not. In the long term, we can design a profiler for profile guided optimizations (PGO). In the short term, we can use hints as type hints or manual hints, to decide which functions should be optimized or not. > On the other hand, I think it would be pretty interesting for the core > python community to come up with source-level constructs that could help > with this sort of thing. For example, one idea is to make it canonical to > do something like: > def f(): > from __builtin__ import len > return len("abc") > > and then let implementations pattern-match this to statically resolve the > import. (...) As written in other messages, they are already many ways to reduce the "overhead" of the Python language. But I'm trying to write an generic optimizer which would not require to modify the source code. Maybe we can modify Python to make some of these optimizations easy to use, but I'm not sure that it's worth it. Victor From victor.stinner at gmail.com Wed Oct 21 23:19:41 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 21 Oct 2015 23:19:41 +0200 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: 2015-10-21 23:03 GMT+02:00 Antoine Pitrou : > I think adding a version number to dicts would be a very cheap change > and would potentially help several CPython-based projects (including > perhaps Numba some day, though nowadays it wouldn't benefit). I tried to leave the dict type unchanged (it's not true in the PoC right now, but I'm working on fixing this). Currently, the versionned dictionary type (fat.verdict) is only used when Python starts in FAT mode and only in a few places: module, class and instance dictionaries. I'm not sure that it's worth to add the versionning feature to the base dict type. It uses a little bit more memory, and IMHO the dict type is already heavy in term of memory usage (compared to a compact C array). I'm trying to keep all FAT features optional to be able to easily compare performances, but also ensure that the FAT mode can be added to CPython without hurting performances (CPU and memory) when it's disabled. > However, I would also like to challenge the idea that an accelerator > has to be 100% compatible. If your API is opt-in (which Numba's is), > then you can happily relax some constraints (provided it is documented) > in exchange for much faster code. Our experience with Numba is that > nobody cares if you can't monkeypatch the len() builtin, for example. Sure, it will be possible to skip some guards if you know that builtins are never monkey-patched, that a module variable is constant, that a class will never be modified, etc. This will reduce even more the cost of guards (especially when all guards are skipped :-)). But I prefer to start with something that doesn't change the Python semantic at all, and then provide opt-in options to give more freedom to the optimizer. Victor From tjreedy at udel.edu Thu Oct 22 03:11:33 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 21 Oct 2015 21:11:33 -0400 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <5627CE63.9030704@mail.de> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <5627CE63.9030704@mail.de> Message-ID: On 10/21/2015 1:41 PM, Sven R. Kunze wrote: > On 21.10.2015 00:50, Chris Angelico wrote: >> She recommends a massive superclass that's capable of any form of injection > > Nope. She does not. > > The "superclass" is not "massive" at all. It is even slimmer as > orthogonal aspects are refactored out into separate entities. In fact, > it makes it even easier to test and maintain these separate aspects (the > core dev should be interested in that). Furthermore, it's, of course, up > to debate which aspects should be injectable and which are not. The dict class itself is, in a sense, a poor example for this discussion. It is a critical part of Python's infrastructure, involved in a large fraction of executed statements. It therefore needs to be as fast as possible. For CPython, this means a heavily optimized C implementation that disallows injection and that takes shortcuts like bypassing method lookups. This makes the usefulness of subclassing limited. Of course, before 2.2, dict (and other built-in types) could not even be subclassed. UserDict is the 'user base dict class', meant to be subclassed by users. A solution to UserDict being too slow could be a C accelerator that did not bypass method lookups. A revised UserDict could be designed for injection in the way being discussed. -- Terry Jan Reedy From yselivanov.ml at gmail.com Thu Oct 22 04:24:02 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 21 Oct 2015 22:24:02 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions Message-ID: <562848C2.4090509@gmail.com> Hi, I was a little bit frustrated that Sublime Text and Atom didn't support all Python 3 features (mainly annotations & async/await syntax), so I decided to write a new highlighter: https://github.com/MagicStack/MagicPython In the process, we had to make a decision on how to highlight raw string literals -- r''. Many existing highlighters assume that all raw strings are regexps, and highlight them as such, i.e. '\s' and '\n' will be highlighted. I think that it might be a good idea to state the following in PEP 8: - use r'...' strings for raw strings that describe regular expressions; these strings might be highlighted specially in some editors. - use R'...' strings for raw strings; editors *should not* highlight any escaped characters in them. What do you think? Yury From ben+python at benfinney.id.au Thu Oct 22 04:44:14 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 22 Oct 2015 13:44:14 +1100 Subject: [Python-ideas] PEP 8: raw strings & regular expressions References: <562848C2.4090509@gmail.com> Message-ID: <858u6vpqpt.fsf@benfinney.id.au> Yury Selivanov writes: > I was a little bit frustrated that Sublime Text and Atom didn't > support all Python 3 features (mainly annotations & async/await > syntax), so I decided to write a new highlighter: Thanks for scratching your itch and releasing the result as free software! > In the process, we had to make a decision on how to highlight raw > string literals -- r''. Many existing highlighters assume that all raw > strings are regexps, and highlight them as such, i.e. '\s' and '\n' > will be highlighted. That is evidently a simple mistake. Merely knowing that a token is a raw string does not justify the assumption that the string is a regular expression, or a filesystem entry name, or a line in a network protocol, or anything except plain text. Perhaps some more explicit context could be used to signal what the intent of a raw string is, but you'd need to find a strong consensus that programmers actually intend that. ?It's a raw string? doesn't justify any of those assumptions. > I think that it might be a good idea to state the following in PEP 8: No, I don't think the mistaken assumptions you've described should be enshrined in a style guide. Instead, the mistaken assumptions should be changed. -- \ ?We have clumsy, sputtering, inefficient brains?. It is a | `\ *struggle* to be rational and objective, and failures are not | _o__) evidence for an alternative reality.? ?Paul Z. Myers, 2010-10-14 | Ben Finney From yselivanov.ml at gmail.com Thu Oct 22 04:53:05 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 21 Oct 2015 22:53:05 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <858u6vpqpt.fsf@benfinney.id.au> References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> Message-ID: <56284F91.8020600@gmail.com> On 2015-10-21 10:44 PM, Ben Finney wrote: >> >In the process, we had to make a decision on how to highlight raw >> >string literals -- r''. Many existing highlighters assume that all raw >> >strings are regexps, and highlight them as such, i.e. '\s' and '\n' >> >will be highlighted. > That is evidently a simple mistake. Merely knowing that a token is a raw > string does not justify the assumption that the string is a regular > expression, or a filesystem entry name, or a line in a network protocol, > or anything except plain text. I agree 100%. But: github, gitlab, Atom, Sublime Text, and many other tools assume that raw strings (with lowercase r) are regexps. If you don't highlight them as such, people think that it's a bug. Since I wanted MagicPython to be a drop-in replacement for standard highlighters, I simply *could not* change this behavior. It's already a standard of some sorts, whether we like it or not. > Perhaps some more explicit context could be used to signal what the > intent of a raw string is, but you'd need to find a strong consensus > that programmers actually intend that. ?It's a raw string? doesn't > justify any of those assumptions. > If we want to design some special marker for highlighters to hint what language is in the string, I'd strongly suggest that it should be before the string literal. For instance, it *won't* be possible for most highlighters to detect this: my_re = '''... ...''' # regex Yury From rymg19 at gmail.com Thu Oct 22 04:58:10 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 21 Oct 2015 21:58:10 -0500 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <56284F91.8020600@gmail.com> References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> <56284F91.8020600@gmail.com> Message-ID: <15E3326A-68CA-42DA-BF2E-F17FDF797721@gmail.com> On October 21, 2015 9:53:05 PM CDT, Yury Selivanov wrote: >On 2015-10-21 10:44 PM, Ben Finney wrote: >>> >In the process, we had to make a decision on how to highlight raw >>> >string literals -- r''. Many existing highlighters assume that all >raw >>> >strings are regexps, and highlight them as such, i.e. '\s' and '\n' >>> >will be highlighted. >> That is evidently a simple mistake. Merely knowing that a token is a >raw >> string does not justify the assumption that the string is a regular >> expression, or a filesystem entry name, or a line in a network >protocol, >> or anything except plain text. > >I agree 100%. > >But: github, gitlab, Atom, Sublime Text, and many other tools GitHub and Atom both use language-python. >assume that raw strings (with lowercase r) are regexps. If you >don't highlight them as such, people think that it's a bug. > >Since I wanted MagicPython to be a drop-in replacement for >standard highlighters, I simply *could not* change this >behavior. It's already a standard of some sorts, whether we >like it or not. > >> Perhaps some more explicit context could be used to signal what the >> intent of a raw string is, but you'd need to find a strong consensus >> that programmers actually intend that. ?It's a raw string? doesn't >> justify any of those assumptions. >> > >If we want to design some special marker for highlighters to >hint what language is in the string, I'd strongly suggest that >it should be before the string literal. For instance, it >*won't* be possible for most highlighters to detect this: > > my_re = '''... > ...''' # regex > >Yury >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From rymg19 at gmail.com Thu Oct 22 04:52:42 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 21 Oct 2015 21:52:42 -0500 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562848C2.4090509@gmail.com> References: <562848C2.4090509@gmail.com> Message-ID: <1F751EE2-ED99-4F21-BE0F-DE1CA0D0E3C4@gmail.com> On October 21, 2015 9:24:02 PM CDT, Yury Selivanov wrote: >Hi, > >I was a little bit frustrated that Sublime Text and Atom didn't support >all Python 3 features (mainly annotations & async/await syntax), so I >decided to write a new highlighter: > > https://github.com/MagicStack/MagicPython > Now see if you can get the Linguist guys to use it! :) Every time I look at the Fbuild source online, my eyes sting due to the lack of annotation support in language-python, which screws up highlighting for a bit: https://github.com/felix-lang/fbuild/blob/master/lib/fbuild/builders/c/gcc/__init__.py#L436 - Keywords aren't highlighted for several lines. Or: https://github.com/felix-lang/fbuild/blob/master/lib/fbuild/builders/bison.py#L15 Where almost (but not quiet) everything is un-highlighted! :O >In the process, we had to make a decision on how to highlight raw >string >literals -- r''. Many existing highlighters assume that all raw >strings >are regexps, and highlight them as such, i.e. '\s' and '\n' will be >highlighted. > >I think that it might be a good idea to state the following in PEP 8: > >- use r'...' strings for raw strings that describe regular expressions; >these strings might be highlighted specially in some editors. > >- use R'...' strings for raw strings; editors *should not* highlight >any escaped characters in them. > >What do you think? > >Yury > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. From yselivanov.ml at gmail.com Thu Oct 22 05:22:05 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 21 Oct 2015 23:22:05 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <15E3326A-68CA-42DA-BF2E-F17FDF797721@gmail.com> References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> <56284F91.8020600@gmail.com> <15E3326A-68CA-42DA-BF2E-F17FDF797721@gmail.com> Message-ID: <5628565D.1010509@gmail.com> On 2015-10-21 10:58 PM, Ryan Gonzalez wrote: > On October 21, 2015 9:53:05 PM CDT, Yury Selivanov wrote: >> >On 2015-10-21 10:44 PM, Ben Finney wrote: >>>>> >>> >In the process, we had to make a decision on how to highlight raw >>>>> >>> >string literals -- r''. Many existing highlighters assume that all >> >raw >>>>> >>> >strings are regexps, and highlight them as such, i.e. '\s' and '\n' >>>>> >>> >will be highlighted. >>> >>That is evidently a simple mistake. Merely knowing that a token is a >> >raw >>> >>string does not justify the assumption that the string is a regular >>> >>expression, or a filesystem entry name, or a line in a network >> >protocol, >>> >>or anything except plain text. >> > >> >I agree 100%. >> > >> >But: github, gitlab, Atom, Sublime Text, and many other tools > GitHub and Atom both use language-python. > Yeah. I even created a PR to use MagicPython in language-python a few days ago. BTW, here's a link to github to show how raw strings are highlighted: https://github.com/python/cpython/blob/master/Lib/_pydecimal.py#L6087 Yury From njs at pobox.com Thu Oct 22 05:31:07 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 21 Oct 2015 20:31:07 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <858u6vpqpt.fsf@benfinney.id.au> References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> Message-ID: On Wed, Oct 21, 2015 at 7:44 PM, Ben Finney wrote: > Yury Selivanov writes: >> In the process, we had to make a decision on how to highlight raw >> string literals -- r''. Many existing highlighters assume that all raw >> strings are regexps, and highlight them as such, i.e. '\s' and '\n' >> will be highlighted. > > That is evidently a simple mistake. Merely knowing that a token is a raw > string does not justify the assumption that the string is a regular > expression, or a filesystem entry name, or a line in a network protocol, > or anything except plain text. This isn't necessarily true, just as a matter of like... epistemology. For example, if hypothetically it turned out that 99% of raw strings are in fact regular expressions, then knowing something is a raw string would give you quite a bit of evidence that it's a regular expression -- quite possibly enough to justify treating it as such for something like code highlighting. I haven't actually gathered any data to find out how strong the association between raw strings and regexen is, but it'd be pretty easy for someone to do. (Parse a large corpus of python code to extract all raw strings, randomly subsample 100 of them, review manually to decide if each is a regex.) -n From yselivanov.ml at gmail.com Thu Oct 22 05:35:18 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 21 Oct 2015 23:35:18 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> Message-ID: <56285976.9090609@gmail.com> On 2015-10-21 11:31 PM, Nathaniel Smith wrote: > I haven't actually gathered any data to find out how strong the > association between raw strings and regexen is, but it'd be pretty > easy for someone to do. (Parse a large corpus of python code to > extract all raw strings, randomly subsample 100 of them, review > manually to decide if each is a regex.) I haven't done this in a scientific way you suggest, but I did glance over the stdlib when I was testing MP. Most of the raw strings I saw were either docstrings or regexps. Docstrings are possible to detect by highlighters, so MagicPython does not highlight r'' as regexp if it's a docstring. Yury From scoutoss at gmail.com Wed Oct 21 20:36:30 2015 From: scoutoss at gmail.com (Scott Sanderson) Date: Wed, 21 Oct 2015 11:36:30 -0700 (PDT) Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: <878u6ww2e3.fsf@fastmail.com> References: <87pp08w4em.fsf@fastmail.com> <878u6ww2e3.fsf@fastmail.com> Message-ID: > > It could just as easily be something like > > def foo(): > return _1(abc) > > foo = add_consts(foo, {'_1':len}) > You can actually do this already in pure Python. We implemented almost exactly what's proposed here as the asconstants decorator in codetransformer : >>> from codetransformer.transformers import asconstants >>> @asconstants(a=1) >>> def f(): ... return a ... >>> f() 1 >>> a = 5 >>> f() 1 This works by making two changes to the function's __code__: 1. We add an entry to foo.__code__.co_consts. 2. We rewrite all LOAD_{GLOBAL|NAME} bytecode instructions of that name to be LOAD_CONSTs instead. This makes the function itself slightly faster, since LOAD_CONST is generally faster than other loads. But more importantly, it statically freezes the value of a particular name in the function, which makes it possible/sane to implement other more interesting transformations/optimizations. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Oct 22 00:30:03 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Oct 2015 11:30:03 +1300 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: <5626B8AB.5040806@oddbird.net> References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> Message-ID: <562811EB.8020103@canterbury.ac.nz> Carl Meyer wrote: > It takes her a couple minutes more to get to the > real point, which starts at the slide "inheritance is for > specialization, not for sharing code." I'm not sure there's any content in that statement. In a duck-typed language, sharing code is the *only* reason to use inheritance. Also, what is "specialisation" anyway? Any time you have two objects with the same interface, and mostly-similar but slightly different behaviours, you could regard one as being a specialisation of the other. Whether they share any code or not is an implementation detail. I think the real issue is that if you specialise something by inheriting and overriding random methods, you make your class fragile with respect to changes in the base class. If the author of the base class changes something undocumented about the way its methods interact with each other, your class may break. With pluggable behaviours, there is a well-defined interface between the main class and the classes representing the behaviours, so that sort of thing is much less likely to happen. You could get the same result with inheritance by designating some methods as "designed to be overridden", and documenting how they interact with the rest of the class. Effectively you are then defining an interface *within* the class and between the class and its subclasses. This is really just another implementation of pluggable behaviours where the plugging mechanism consists of overriding one or more methods. So to summarise, I would say that the issue here isn't really composition vs. inheritance, but structured vs. ad-hoc behaviour modification. I wouldn't say that one is always better than the other. Sometimes you need to do things that the author of a class didn't anticipate, and then the ability to override things in an ad-hoc manner is very useful. But I think the speaker is right to point out that doing *everything* that way can lead to a lot of trouble. -- Greg From ben+python at benfinney.id.au Thu Oct 22 07:44:00 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 22 Oct 2015 16:44:00 +1100 Subject: [Python-ideas] PEP 8: raw strings & regular expressions References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> Message-ID: <854mhjpie7.fsf@benfinney.id.au> Nathaniel Smith writes: > On Wed, Oct 21, 2015 at 7:44 PM, Ben Finney wrote: > > [Automatically interpreting every raw string as a regex pattern] is > > evidently a simple mistake. Merely knowing that a token is a raw > > string does not justify the assumption that the string is a regular > > expression, or a filesystem entry name, or a line in a network > > protocol, or anything except plain text. > > This isn't necessarily true, just as a matter of like... epistemology. Well, yes, if you like. Epistemically, a sytntax highlighter cannot know that a raw string is, merely because it's a raw string, definitely a regular expression pattern. We have a definition of the language which allows syntax highlighters to know with certainty what is and is not a particular element of the language. So if the highlighter shows a sequence of characters as being what the Python language definition says it is, then it will not be wrong in any case. We do not have a definition which allows syntax highlighters to decide that a raw string is or is not a regular expression, merely because it's a raw string. So if a highlighter shows a sequence of characters in a Python program as being a regular expression, it will be wrong for some cases. > For example, if hypothetically it turned out that 99% of raw strings > are in fact regular expressions, then knowing something is a raw > string would give you quite a bit of evidence that it's a regular > expression -- quite possibly enough to justify treating it as such for > something like code highlighting. Presenting the code highlighted to show particular semantics is a binary state: it either is shown as (for example) a regular expression, or it is not. The reader only gets to see what the highlighter decided, not how certain the epistemic decision was. How is the person viewing it to know whether the highlighter is wrong about the intention of the code in any particular case, or if the highlighter is right and the code doesn't match the author's intention? If the reader has to second-guess the highlighter (am I wrong here, or is the highlighter wrong, or both?) every time it doesn't match expectations, that's a poor syntax highlighter which should never have made such a binary decision on uncertain data. -- \ ?Never express yourself more clearly than you are able to | `\ think.? ?Niels Bohr | _o__) | Ben Finney From tjreedy at udel.edu Thu Oct 22 09:44:58 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 22 Oct 2015 03:44:58 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562848C2.4090509@gmail.com> References: <562848C2.4090509@gmail.com> Message-ID: On 10/21/2015 10:24 PM, Yury Selivanov wrote: > Hi, > > I was a little bit frustrated that Sublime Text and Atom didn't support > all Python 3 features (mainly annotations & async/await syntax), so I > decided to write a new highlighter: > > https://github.com/MagicStack/MagicPython > > In the process, we had to make a decision on how to highlight raw string > literals -- r''. Many existing highlighters assume that all raw strings > are regexps, and highlight them as such, i.e. '\s' and '\n' will be > highlighted. What 3rd parth editors do is their business. > I think that it might be a good idea to state the following in PEP 8: > > - use r'...' strings for raw strings that describe regular expressions; > these strings might be highlighted specially in some editors. > > - use R'...' strings for raw strings; editors *should not* highlight > any escaped characters in them. > > What do you think? I think it a bad idea. For beginners on Windows, r'windows\path\file.py' might be more common than r're'. I have never seen R used. If you wanted to promote the use of the currently rare R for REs, and have editors specially mark raw literals with this special prefix, I would not mind. -- Terry Jan Reedy From mal at egenix.com Thu Oct 22 10:32:43 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 22 Oct 2015 10:32:43 +0200 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <858u6vpqpt.fsf@benfinney.id.au> References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> Message-ID: <56289F2B.6010808@egenix.com> On 22.10.2015 04:44, Ben Finney wrote: > Yury Selivanov > writes: > >> I was a little bit frustrated that Sublime Text and Atom didn't >> support all Python 3 features (mainly annotations & async/await >> syntax), so I decided to write a new highlighter: > > Thanks for scratching your itch and releasing the result as free software! > >> In the process, we had to make a decision on how to highlight raw >> string literals -- r''. Many existing highlighters assume that all raw >> strings are regexps, and highlight them as such, i.e. '\s' and '\n' >> will be highlighted. > > That is evidently a simple mistake. Merely knowing that a token is a raw > string does not justify the assumption that the string is a regular > expression, or a filesystem entry name, or a line in a network protocol, > or anything except plain text. > > Perhaps some more explicit context could be used to signal what the > intent of a raw string is, but you'd need to find a strong consensus > that programmers actually intend that. ?It's a raw string? doesn't > justify any of those assumptions. > >> I think that it might be a good idea to state the following in PEP 8: > > No, I don't think the mistaken assumptions you've described should be > enshrined in a style guide. Instead, the mistaken assumptions should be > changed. Agreed. Highlighters should follow language definitions, not the other way around ;-) Yuri: Perhaps you could make the behavior optional in your highlighter and allow people to turn off the highlighting as regular expression, if they find they don't like the highlighting. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 22 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rob.cliffe at btinternet.com Thu Oct 22 11:41:44 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 22 Oct 2015 10:41:44 +0100 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> Message-ID: <5628AF58.1010001@btinternet.com> On 22/10/2015 08:44, Terry Reedy wrote: > On 10/21/2015 10:24 PM, Yury Selivanov wrote: >> Hi, >> >> I was a little bit frustrated that Sublime Text and Atom didn't support >> all Python 3 features (mainly annotations & async/await syntax), so I >> decided to write a new highlighter: >> >> https://github.com/MagicStack/MagicPython >> >> In the process, we had to make a decision on how to highlight raw string >> literals -- r''. Many existing highlighters assume that all raw strings >> are regexps, and highlight them as such, i.e. '\s' and '\n' will be >> highlighted. > > What 3rd parth editors do is their business. > >> I think that it might be a good idea to state the following in PEP 8: >> >> - use r'...' strings for raw strings that describe regular expressions; >> these strings might be highlighted specially in some editors. >> >> - use R'...' strings for raw strings; editors *should not* highlight >> any escaped characters in them. >> >> What do you think? > > I think it a bad idea. For beginners on Windows, > r'windows\path\file.py' might be more common than r're'. I have never > seen R used. > > If you wanted to promote the use of the currently rare R for REs, and > have editors specially mark raw literals with this special prefix, I > would not mind. > +1. I may or may not be typical, but I use regular expressions very rarely. Rob Cliffe. From abarnert at yahoo.com Thu Oct 22 12:00:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 22 Oct 2015 03:00:20 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> Message-ID: <2302BDF8-74D6-4752-9182-7856F07F1A24@yahoo.com> On Oct 22, 2015, at 00:44, Terry Reedy wrote: > >> On 10/21/2015 10:24 PM, Yury Selivanov wrote: > >> I think that it might be a good idea to state the following in PEP 8: >> >> - use r'...' strings for raw strings that describe regular expressions; >> these strings might be highlighted specially in some editors. >> >> - use R'...' strings for raw strings; editors *should not* highlight >> any escaped characters in them. >> >> What do you think? > > I think it a bad idea. For beginners on Windows, r'windows\path\file.py' might be more common than r're'. It's also worth noting that an awful lot of code that uses Windows pathnames is either beginner code, local scripts, or closed-source commercial code, which means a typical code search is probably going to vastly underrepresent how common they are in raw strings. (Of course that same fact means it may be perfectly reasonable for GitHub to assume raw strings are regexps rather than Windows pathnames, even if it isn't reasonable for Python itself, or general-purpose tools like IDLE?) > I have never seen R used. > > If you wanted to promote the use of the currently rare R for REs, and have editors specially mark raw literals with this special prefix, I would not mind. That doesn't sound as bad. But I still don't like it. Where else does Python provide two equivalent ways to do something, specifically to support external semantic connotations? It's like having <> and != both mean the same thing to support people coming up with some language-external difference between the spellings. (Yes, I realize there are a few cases like this?e.g., someone could use the fact that int and 'int' annotate the same type to give them different connotations?but those are accidental effects of some other language feature; PEP 8 certainly isn't going to suggest using 'int' to mean one thing and int another.) If we really want there to be a difference, we should have a regex literal syntax?maybe an x or s prefix or something?in place of re.compile(r'?'). From storchaka at gmail.com Thu Oct 22 12:14:51 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 22 Oct 2015 13:14:51 +0300 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562848C2.4090509@gmail.com> References: <562848C2.4090509@gmail.com> Message-ID: On 22.10.15 05:24, Yury Selivanov wrote: > I was a little bit frustrated that Sublime Text and Atom didn't support > all Python 3 features (mainly annotations & async/await syntax), so I > decided to write a new highlighter: > > https://github.com/MagicStack/MagicPython > > In the process, we had to make a decision on how to highlight raw string > literals -- r''. Many existing highlighters assume that all raw strings > are regexps, and highlight them as such, i.e. '\s' and '\n' will be > highlighted. > > I think that it might be a good idea to state the following in PEP 8: > > - use r'...' strings for raw strings that describe regular expressions; > these strings might be highlighted specially in some editors. > > - use R'...' strings for raw strings; editors *should not* highlight > any escaped characters in them. > > What do you think? Do highlighters highlight formatting strings ("%*.2f" or "{0.addr:0{}x}") specially? From rosuav at gmail.com Thu Oct 22 12:52:52 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 22 Oct 2015 21:52:52 +1100 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <2302BDF8-74D6-4752-9182-7856F07F1A24@yahoo.com> References: <562848C2.4090509@gmail.com> <2302BDF8-74D6-4752-9182-7856F07F1A24@yahoo.com> Message-ID: On Thu, Oct 22, 2015 at 9:00 PM, Andrew Barnert via Python-ideas wrote: > But I still don't like it. Where else does Python provide two equivalent ways to do something, specifically to support external semantic connotations? It's like having <> and != both mean the same thing to support people coming up with some language-external difference between the spellings. > I don't know, but since PEP 8 recommends using """ for docstrings and not ''', it would be entirely possible for someone to assign special meaning to the use of ''' for a docstring. ChrisA From geoffspear at gmail.com Thu Oct 22 14:19:46 2015 From: geoffspear at gmail.com (Geoffrey Spear) Date: Thu, 22 Oct 2015 08:19:46 -0400 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: On Wed, Oct 21, 2015 at 4:09 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Wed, Oct 21, 2015 at 3:49 PM, Geoffrey Spear > wrote: > >> "We stole this short spelling from C" shouldn't be an argument in favor >> of any spelling, in my opinion. > > > Can you explain your opinion? C is by far the most influential language > in the history of computing. As a source of keywords it is second only to > English. > Lots of stuff in C favors terseness over readability. Python generally takes the opposite approach to naming things. Obviously this is sometimes a cause of confusion to people new to the language, when people think "oh, Python is just like English, so I should use 'is' instead of '=='" or "why doesn't `if foo or bar == 'baz'` do what I mean?", but for the most part having easy-to-read actual English words for keywords and functions with names that actually say what they do instead of making some obscure reference to a badly-named C library function written at a time when limiting identifiers to 6 letters seemed like a good idea to save precious disk space (or room on a punchcard), in my opinion, has greatly contributed to Python's reputation as a readable language. "elif" is a wart, but again, one I think we're stuck with. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Thu Oct 22 16:04:15 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 22 Oct 2015 10:04:15 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> Message-ID: <5628ECDF.7000702@gmail.com> On 2015-10-22 6:14 AM, Serhiy Storchaka wrote: > On 22.10.15 05:24, Yury Selivanov wrote: >> I was a little bit frustrated that Sublime Text and Atom didn't support >> all Python 3 features (mainly annotations & async/await syntax), so I >> decided to write a new highlighter: >> >> https://github.com/MagicStack/MagicPython >> >> In the process, we had to make a decision on how to highlight raw string >> literals -- r''. Many existing highlighters assume that all raw strings >> are regexps, and highlight them as such, i.e. '\s' and '\n' will be >> highlighted. >> >> I think that it might be a good idea to state the following in PEP 8: >> >> - use r'...' strings for raw strings that describe regular expressions; >> these strings might be highlighted specially in some editors. >> >> - use R'...' strings for raw strings; editors *should not* highlight >> any escaped characters in them. >> >> What do you think? > > Do highlighters highlight formatting strings ("%*.2f" or > "{0.addr:0{}x}") specially? Many highlight old-style formatting. Less highlight the new-style. Github, for example, highlights both. Yury From storchaka at gmail.com Thu Oct 22 18:03:05 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 22 Oct 2015 19:03:05 +0300 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <5628ECDF.7000702@gmail.com> References: <562848C2.4090509@gmail.com> <5628ECDF.7000702@gmail.com> Message-ID: On 22.10.15 17:04, Yury Selivanov wrote: > On 2015-10-22 6:14 AM, Serhiy Storchaka wrote: >> Do highlighters highlight formatting strings ("%*.2f" or >> "{0.addr:0{}x}") specially? > > Many highlight old-style formatting. Less highlight the new-style. > > Github, for example, highlights both. If the fact that not all strings are formatting strings is ignored, you are free to ignore the fact that not all raw strings are regular expression patterns. From srkunze at mail.de Thu Oct 22 18:09:04 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 22 Oct 2015 18:09:04 +0200 Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict In-Reply-To: References: <5620D33A.3050709@feete.org> <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com> <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com> <56268487.9050405@mail.de> <5626B8AB.5040806@oddbird.net> <5627CE63.9030704@mail.de> Message-ID: <56290A20.2090602@mail.de> On 22.10.2015 03:11, Terry Reedy wrote: > On 10/21/2015 1:41 PM, Sven R. Kunze wrote: >> The "superclass" is not "massive" at all. It is even slimmer as >> orthogonal aspects are refactored out into separate entities. In fact, >> it makes it even easier to test and maintain these separate aspects (the >> core dev should be interested in that). Furthermore, it's, of course, up >> to debate which aspects should be injectable and which are not. > > The dict class itself is, in a sense, a poor example for this > discussion. It is a critical part of Python's infrastructure, > involved in a large fraction of executed statements. It therefore > needs to be as fast as possible. For CPython, this means a heavily > optimized C implementation that disallows injection and that takes > shortcuts like bypassing method lookups. This makes the usefulness of > subclassing limited. The discussion *is* about dict: normal_dict = dict() ordered_dict = dict(order=dict.order_by_insert) sorted_dict = dict(order=sorted) sorted_default_dict = dict(order=sorted, default=int) Why couldn't dict() or{} redirect to the super-fast built-in C-implementation whereas dict(order=sorted, default=int) redirects to some more feature rich one? As you see, I for one don't see a contradiction between performance and features. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Oct 22 18:13:38 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Oct 2015 09:13:38 -0700 Subject: [Python-ideas] PEP-505: Draft (Coalescing Operators) In-Reply-To: References: Message-ID: Hi Mark, I'm confused by the example >>> 2 or None ?? err() The PEP states this raises, but since 2 is already True, and you state that the operator has higher precedence than 'or', this example shouldn't evaluate anything to the right of 'or', so it shouldn't raise -- just like "2 or err()" doesn't raise. Also, unrelated (but in the same section) I think this operator should have a precedence higher than 'not'. That is, >>> not x ?? y should be parsed as >>> not (x ?? y) After all, if we parsed it as >>> (not x) ?? y it wouldn't be very useful, since (not x) never returns None, hence y would never be evaluated here -- this is a redundant way to spell (not x). Other notes: "The idea of a None -aware function invocation syntax was discussed on python- ideas, but the idea was rejected by BDFL." -- I don't think it's that strong. It's merely that the use cases for x?.attr and x?[key] are stronger, and various people suggested to focus on the most important. "[...] except it returns its left operand if that operand is None and otherwise returns the right operand." -- the part about returning the left operand if it is None is silly -- there is only one None, so it should just say "returns None if the left operand is None and otherwise returns the right operand." I'll have to think more about the rest. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu Oct 22 18:19:20 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 22 Oct 2015 18:19:20 +0200 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562848C2.4090509@gmail.com> References: <562848C2.4090509@gmail.com> Message-ID: <56290C88.9040302@mail.de> On 22.10.2015 04:24, Yury Selivanov wrote: > Hi, > > I was a little bit frustrated that Sublime Text and Atom didn't support > all Python 3 features (mainly annotations & async/await syntax), so I > decided to write a new highlighter: > > https://github.com/MagicStack/MagicPython > > In the process, we had to make a decision on how to highlight raw string > literals -- r''. Many existing highlighters assume that all raw strings > are regexps, and highlight them as such, i.e. '\s' and '\n' will be > highlighted. > > I think that it might be a good idea to state the following in PEP 8: > > - use r'...' strings for raw strings that describe regular expressions; > these strings might be highlighted specially in some editors. > > - use R'...' strings for raw strings; editors *should not* highlight > any escaped characters in them. > > What do you think? > > Yury Thanks releasing your package. I think: leave it as is. Why? 1) It's not really something that needs a standard. 2) r and R are equivalent. Best, Sven From srkunze at mail.de Thu Oct 22 18:22:31 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 22 Oct 2015 18:22:31 +0200 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> Message-ID: <56290D47.9030800@mail.de> On 22.10.2015 14:19, Geoffrey Spear wrote: > Obviously this is sometimes a cause of confusion to people new to the > language, when people think "oh, Python is just like English, so I > should use 'is' instead of '=='" or "why doesn't `if foo or bar == > 'baz'` do what I mean?", but for the most part having easy-to-read > actual English words for keywords and functions with names that > actually say what they do instead of making some obscure reference to > a badly-named C library function written at a time when limiting > identifiers to 6 letters seemed like a good idea to save precious disk > space (or room on a punchcard), in my opinion, has greatly contributed > to Python's reputation as a readable language. So, why stop there? ;) > "elif" is a wart, but again, one I think we're stuck with. Dangerous thinking. I don't regard changing it to be a big deal given 10 years or so of parallel usage. Best, Sven From mehaase at gmail.com Thu Oct 22 18:36:07 2015 From: mehaase at gmail.com (Mark E. Haase) Date: Thu, 22 Oct 2015 12:36:07 -0400 Subject: [Python-ideas] PEP-505: Draft (Coalescing Operators) In-Reply-To: References: Message-ID: Yes, that `or` example is incorrect. I have short circuiting and precedence jumbled up in my head, but thinking about that example has improved my understanding. I will fix that and incorporate your other notes shortly. Thanks! On Thu, Oct 22, 2015 at 12:13 PM, Guido van Rossum wrote: > Hi Mark, > > I'm confused by the example > > >>> 2 or None ?? err() > > The PEP states this raises, but since 2 is already True, and you state > that the operator has higher precedence than 'or', this example shouldn't > evaluate anything to the right of 'or', so it shouldn't raise -- just like > "2 or err()" doesn't raise. > > Also, unrelated (but in the same section) I think this operator should > have a precedence higher than 'not'. That is, > > >>> not x ?? y > > should be parsed as > > >>> not (x ?? y) > > After all, if we parsed it as > > >>> (not x) ?? y > > it wouldn't be very useful, since (not x) never returns None, hence y > would never be evaluated here -- this is a redundant way to spell (not x). > > Other notes: > > "The idea of a None -aware function invocation syntax was discussed on > python- ideas, but the idea was rejected by BDFL." -- I don't think it's > that strong. It's merely that the use cases for x?.attr and x?[key] are > stronger, and various people suggested to focus on the most important. > > "[...] except it returns its left operand if that operand is None and > otherwise returns the right operand." -- the part about returning the left > operand if it is None is silly -- there is only one None, so it should just > say "returns None if the left operand is None and otherwise returns the > right operand." > > I'll have to think more about the rest. > > -- > --Guido van Rossum (python.org/~guido) > -- Mark E. Haase 202-815-0201 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Oct 22 19:24:34 2015 From: brett at python.org (Brett Cannon) Date: Thu, 22 Oct 2015 17:24:34 +0000 Subject: [Python-ideas] Add specialized bytecode with guards to functions In-Reply-To: References: Message-ID: On Wed, 21 Oct 2015 at 14:20 Victor Stinner wrote: > 2015-10-21 23:03 GMT+02:00 Antoine Pitrou : > > I think adding a version number to dicts would be a very cheap change > > and would potentially help several CPython-based projects (including > > perhaps Numba some day, though nowadays it wouldn't benefit). > > I tried to leave the dict type unchanged (it's not true in the PoC > right now, but I'm working on fixing this). Currently, the versionned > dictionary type (fat.verdict) is only used when Python starts in FAT > mode and only in a few places: module, class and instance > dictionaries. I'm not sure that it's worth to add the versionning > feature to the base dict type. It uses a little bit more memory, and > IMHO the dict type is already heavy in term of memory usage (compared > to a compact C array). > I'm not sure if Antoine was truly proposing adding version number support to all dicts, but I know I wasn't. I would propose creating a dict subclass and use it in exactly the places you outlined, Victor: modules, classes, and instance dictionaries (and I would honestly be fine with doing it piecemeal in that order until we have experience as to how truly dynamic people treat instances compared to modules and classes). I would start with a global counter for every time __setitem__() is successful and see where that gets us (both in terms of performance and optimizations). > > I'm trying to keep all FAT features optional to be able to easily > compare performances, but also ensure that the FAT mode can be added > to CPython without hurting performances (CPU and memory) when it's > disabled. > > > However, I would also like to challenge the idea that an accelerator > > has to be 100% compatible. If your API is opt-in (which Numba's is), > > then you can happily relax some constraints (provided it is documented) > > in exchange for much faster code. Our experience with Numba is that > > nobody cares if you can't monkeypatch the len() builtin, for example. > > Sure, it will be possible to skip some guards if you know that > builtins are never monkey-patched, that a module variable is constant, > that a class will never be modified, etc. This will reduce even more > the cost of guards (especially when all guards are skipped :-)). > > But I prefer to start with something that doesn't change the Python > semantic at all, and then provide opt-in options to give more freedom > to the optimizer. > I agree that not all optimizations/accelerators need to be 100% compatible. Numba is in a certain kind of position, though, in that it is applied at the function/method level while what Victor and I are working on applies to all code. So it's just differing levels of granularity which influences how much one feels like they can break the rules in the name of performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Thu Oct 22 19:33:26 2015 From: antoine at python.org (Antoine Pitrou) Date: Thu, 22 Oct 2015 17:33:26 +0000 (UTC) Subject: [Python-ideas] Add specialized bytecode with guards to functions References: Message-ID: Brett Cannon writes: > > I'm not sure if Antoine was truly proposing adding version number > support to all dicts, but I know I wasn't. I am. An empty dict is currently 288 bytes on 64-bit Linux. Even an empty dict with key sharing (e.g. an instance dict, which would be versioned anyway in your proposal) is 96 bytes long. Adding a size_t-sized field would not make a significant difference in memory consumption, and is probably much simpler than ensuring you use the right dict subtype at each place, add some type-checking to all PyDict concrete APIs, etc. And incrementing the version number is cheaper if you don't have a conditional branch based on whether the dict is a versioned or non-versioned dict. Regards Antoine. From srkunze at mail.de Fri Oct 23 09:47:23 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 23 Oct 2015 15:47:23 +0200 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562969B4.5080406@gmail.com> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> Message-ID: <562A3A6B.1060207@mail.de> I think you shouldn't push it, Yury. Most people here seem not to like it that much for many reasons. Best, Sven On 23.10.2015 00:56, Yury Selivanov wrote: > Hi Barry, > > On 2015-10-22 5:17 PM, Barry Warsaw wrote: >>> I think that it might be a good idea to state the following in PEP 8: >>> > >>> >- use r'...' strings for raw strings that describe regular >>> expressions; >>> >these strings might be highlighted specially in some editors. >>> > >>> >- use R'...' strings for raw strings; editors*should not* highlight >>> >any escaped characters in them. >> Aside from other comments in this thread, I think it's generally a >> bad idea to >> codify semantics where there are none defined by the language. So r'' >> and R'' >> are equivalent. It would be like assigning different semantics or >> syntax >> highlighting for 'foo' and "foo". > > The thing is that *a lot* of Python users have already made > the choice -- many tools (including GitHub) do highlight > r'' strings as regexps, and people love this feature, and > complain when it's not implemented in their tool of choice. > This is an extremely useful feature, since any regexp longer > than 20 characters is unreadable at a glance. > > On the other hand, *a lot* of users are annoyed, when their > raw strings are highlighted as regexps (why \n is highlighted > as escaped?). Often that results not in just wrongly > highlighted raw strings, but in broken highlighting of > the whole source file [1]. > > In principle, there is no reason why *both* of these groups > of users can't use one tool and be happy. I propose to > establish a convention in PEP 8, explaining that, while both > literals are semantically equivalent, > > - r'..' strings *should* be used for regexps, > > - R'..' strings *should* be used for unstyled raw strings, > > and tools *should* treat them as such. > > All of this is merely about codifying the current status quo. > > Thanks, > Yury > > > [1] https://gist.github.com/1st1/d3da5d69a9b6c088c26f > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From eric at trueblade.com Sun Oct 25 17:06:16 2015 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 25 Oct 2015 17:06:16 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562969B4.5080406@gmail.com> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> Message-ID: <562D4448.6090208@trueblade.com> On 10/22/2015 06:56 PM, Yury Selivanov wrote: > On the other hand, *a lot* of users are annoyed, when their > raw strings are highlighted as regexps (why \n is highlighted > as escaped?). Often that results not in just wrongly > highlighted raw strings, but in broken highlighting of > the whole source file [1]. ... > [1] https://gist.github.com/1st1/d3da5d69a9b6c088c26f I don't see how this matters to the subject at hand. No matter what you think an r or R string contains, not detecting the end of the string is a failure of the syntax highlighter. It's failing to find the end of the triple-quoted r"""[""" token in this case. Eric. From srkunze at mail.de Sat Oct 24 04:39:59 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Sat, 24 Oct 2015 10:39:59 +0200 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> Message-ID: <562B43DF.9030809@mail.de> On 23.10.2015 18:09, Nick Coghlan wrote: > It's a big deal due to opportunity cost - any time that is spent on > making a second spelling of "elif" possible is time not spent working > on something that directly improves developer productivity (like > format strings) or application correctness (like gradual typing > support). 1) I may re-iterate. Such thinking is dangerous. It could mean standstill. It prevents thinking and progress. That is python-ideas; not python-controlling. 2) Is Python not an open-source project? Could not anybody contribute? To me that means, the sky is the limit. 3) Well, that the features you mentioned really do what they promise can be only be proved if they are implemented and used. So, we will see how they fare. Same, btw., would be for "else if". All of your and mine great thoughts are just that: thoughts, assumptions, arguments. Not facts. > Time spent on frivolous changes also (justifiably) annoys > users of the language given the large numbers of known problems we've > yet to figure out how to find the time and resources to resolve. Cf. above. Okay, enough of resource-thinking. Back to topic. > From a learning perspective, remembering that "elif" is an > abbreviation of "else if" in Python isn't any more complicated than > remembering that "don't" is an abbreviation of "do not" in English. > It's certainly easier to remember than what "def" or "class" means. Let's remember that. > From a readability perspective, inserting an "se " in the middle of > every elif clause doesn't really improve things: > > if x == 1: > ... > elif x == 2: > ... > elif x == 3: > ... > else: > ... > > vs: > > if x == 1: > ... > else if x == 2: > ... > else if x == 3: > ... > else: > ... > > It can actually be argued that it's *harder* to read, since the > typical English phrasing for this kind of formulation is more along > the lines of "X if case A, else Y if case B, else Z", which is the way > conditional expressions are written. I thought "elif" is an abbreviation of "else if" as is "don't" to "do not". So, why are you arguing with differing semantics when it comes to readability? I for one think it improves things (as does several other people on the list), that is why it is on the list. ;-) Why you may ask? Because it just reads like an English sentence: "If that condition, then (==colon) do this, else if another condition, then (==colon) do that, etc. etc." So, we got "if", ":", and "else", this is just all we need to learn. Something, I would be interested in is: what was the reason for introducing the non-English keyword "elif"? Parsing difficulties? Best, Sven From abarnert at yahoo.com Sun Oct 25 17:49:57 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 25 Oct 2015 14:49:57 -0700 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <562B43DF.9030809@mail.de> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: On Oct 24, 2015, at 01:39, Sven R. Kunze wrote: > > I for one think it improves things (as does several other people on the list), that is why it is on the list. ;-) > Why you may ask? Because it just reads like an English sentence: "If that condition, then (==colon) do this, else if another condition, then (==colon) do that, etc. etc." Nobody has ever spoken a sentence in that form. Maybe with "otherwise", even more likely with no connective at all (which you'd spell with a semicolon), but never with "else". That's what's wrong with this whole argument: it's an attempt to make Python "more like English" in a way that's not actually like English. > Something, I would be interested in is: what was the reason for introducing the non-English keyword "elif"? Parsing difficulties? Think about the way "else if" works in C and its descendants and influencees: it's just an "else" clause whose body happens to be an "if" statement. By convention, you don't wrap the "if" statement in braces or indent it an extra level, but there's no syntactic reason not to do so. (And formatter tools need a special case to make them not do so.) In Python, indentation is not optional or conventional, it's syntactic. So the if would have to be indented under the else. Either that, or it would have to work differently from the way it works in all those other languages. Of course that's possible, but that demolishes the other argument for this change: to make Python more like other languages, we'd make it misleadingly appear to be like all those other languages, when it actually works differently. Which would be a great way to confuse people coming to Python from Java, or alternating Python. C, and JS code in their work, etc. From ncoghlan at gmail.com Fri Oct 23 12:09:26 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2015 18:09:26 +0200 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <56290D47.9030800@mail.de> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> Message-ID: On 22 October 2015 at 18:22, Sven R. Kunze wrote: > On 22.10.2015 14:19, Geoffrey Spear wrote: >> >> Obviously this is sometimes a cause of confusion to people new to the >> language, when people think "oh, Python is just like English, so I should >> use 'is' instead of '=='" or "why doesn't `if foo or bar == 'baz'` do what I >> mean?", but for the most part having easy-to-read actual English words for >> keywords and functions with names that actually say what they do instead of >> making some obscure reference to a badly-named C library function written at >> a time when limiting identifiers to 6 letters seemed like a good idea to >> save precious disk space (or room on a punchcard), in my opinion, has >> greatly contributed to Python's reputation as a readable language. > > > So, why stop there? ;) > >> "elif" is a wart, but again, one I think we're stuck with. > > > Dangerous thinking. > > I don't regard changing it to be a big deal given 10 years or so of parallel > usage. It's a big deal due to opportunity cost - any time that is spent on making a second spelling of "elif" possible is time not spent working on something that directly improves developer productivity (like format strings) or application correctness (like gradual typing support). Time spent on frivolous changes also (justifiably) annoys users of the language given the large numbers of known problems we've yet to figure out how to find the time and resources to resolve. >From a learning perspective, remembering that "elif" is an abbreviation of "else if" in Python isn't any more complicated than remembering that "don't" is an abbreviation of "do not" in English. It's certainly easier to remember than what "def" or "class" means. >From a readability perspective, inserting an "se " in the middle of every elif clause doesn't really improve things: if x == 1: ... elif x == 2: ... elif x == 3: ... else: ... vs: if x == 1: ... else if x == 2: ... else if x == 3: ... else: ... It can actually be argued that it's *harder* to read, since the typical English phrasing for this kind of formulation is more along the lines of "X if case A, else Y if case B, else Z", which is the way conditional expressions are written. By contrast, the statement form has an implied "then" after each condition: "if case A, then X, else if case B, then Y, else Z". Compared to dropping "then" entirely, abbreviating "else if" to "elif" is a fairly minor change. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rymg19 at gmail.com Sun Oct 25 19:03:12 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 25 Oct 2015 18:03:12 -0500 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <562B43DF.9030809@mail.de> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: On Sat, Oct 24, 2015 at 3:39 AM, Sven R. Kunze wrote: > On 23.10.2015 18:09, Nick Coghlan wrote: > >> It's a big deal due to opportunity cost - any time that is spent on >> making a second spelling of "elif" possible is time not spent working >> on something that directly improves developer productivity (like >> format strings) or application correctness (like gradual typing >> support). >> > > 1) I may re-iterate. Such thinking is dangerous. It could mean standstill. > It prevents thinking and progress. That is python-ideas; not > python-controlling. > > 2) Is Python not an open-source project? Could not anybody contribute? To > me that means, the sky is the limit. > > 3) Well, that the features you mentioned really do what they promise can > be only be proved if they are implemented and used. So, we will see how > they fare. Same, btw., would be for "else if". All of your and mine great > thoughts are just that: thoughts, assumptions, arguments. Not facts. > > Time spent on frivolous changes also (justifiably) annoys >> users of the language given the large numbers of known problems we've >> yet to figure out how to find the time and resources to resolve. >> > > Cf. above. > > Okay, enough of resource-thinking. Back to topic. > > From a learning perspective, remembering that "elif" is an >> abbreviation of "else if" in Python isn't any more complicated than >> remembering that "don't" is an abbreviation of "do not" in English. >> It's certainly easier to remember than what "def" or "class" means. >> > > Let's remember that. > > From a readability perspective, inserting an "se " in the middle of >> every elif clause doesn't really improve things: >> >> if x == 1: >> ... >> elif x == 2: >> ... >> elif x == 3: >> ... >> else: >> ... >> >> vs: >> >> if x == 1: >> ... >> else if x == 2: >> ... >> else if x == 3: >> ... >> else: >> ... >> >> It can actually be argued that it's *harder* to read, since the >> typical English phrasing for this kind of formulation is more along >> the lines of "X if case A, else Y if case B, else Z", which is the way >> conditional expressions are written. >> > > I thought "elif" is an abbreviation of "else if" as is "don't" to "do > not". So, why are you arguing with differing semantics when it comes to > readability? > > I for one think it improves things (as does several other people on the > list), that is why it is on the list. ;-) > Why you may ask? Because it just reads like an English sentence: "If that > condition, then (==colon) do this, else if another condition, then > (==colon) do that, etc. etc." So, we got "if", ":", and "else", this is > just all we need to learn. > > Two things: 1. That doesn't read like an English sentence. It sounds like an English sentence spoken by someone who's been through a 48-hour C hackathon. 2. Well, elif is used almost everywhere, so 99% of the people who learn Python will have to learn 'elif' the day after 'else if'. > > Something, I would be interested in is: what was the reason for > introducing the non-English keyword "elif"? Parsing difficulties? > > > Best, > Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Oct 25 18:16:34 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 26 Oct 2015 09:16:34 +1100 Subject: [Python-ideas] =?utf-8?q?Limits_to_ideas_for_proposal_in_?= =?utf-8?b?4oCYcHl0aG9uLWlkZWFz4oCZICh3YXM6ICJlbHNlIGlmIiBhcyBlcXVpdmFs?= =?utf-8?b?ZW50IGZvciAiZWxpZiIp?= References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: <858u6qoapp.fsf_-_@benfinney.id.au> "Sven R. Kunze" writes: > 2) Is Python not an open-source project? Could not anybody contribute? > To me that means, the sky is the limit. Easy for you to say, but that's not the limit. The limit in *this* forum is the finite attention of the Python core developers. If you want to discuss sky-is-the-limit ideas, use ?python-list? which is a forum for all topics Python-related. -- \ ?Repetition leads to boredom, boredom to horrifying mistakes, | `\ horrifying mistakes to God-I-wish-I-was-still-bored, and it | _o__) goes downhill from there.? ?Will Larson, 2008-11-04 | Ben Finney From barry at python.org Fri Oct 23 14:18:57 2015 From: barry at python.org (Barry Warsaw) Date: Fri, 23 Oct 2015 14:18:57 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> Message-ID: <20151023141857.0aec87e9@anarchist.wooz.org> On Oct 22, 2015, at 06:56 PM, Yury Selivanov wrote: >In principle, there is no reason why *both* of these groups >of users can't use one tool and be happy. I propose to >establish a convention in PEP 8, explaining that, while both >literals are semantically equivalent, > >- r'..' strings *should* be used for regexps, > >- R'..' strings *should* be used for unstyled raw strings, > >and tools *should* treat them as such. > >All of this is merely about codifying the current status quo. What about all the regexps that are written without raw strings? Won't PEP 8 codification shame everyone into turn them into raw strings when they really don't need to? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sun Oct 25 21:02:22 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Oct 2015 12:02:22 +1100 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <562B43DF.9030809@mail.de> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: On Sat, Oct 24, 2015 at 7:39 PM, Sven R. Kunze wrote: > 1) I may re-iterate. Such thinking is dangerous. It could mean standstill. > It prevents thinking and progress. That is python-ideas; not > python-controlling. The point of discussing ideas is to sort out the good ones from the bad ones. One of the most effective ways to do this is to have someone advocate an idea, and someone else criticize it, with the resulting discussion being (hopefully!) a fruitful analysis of the idea's qualities. So while you're correct that this is called "python-ideas" for a reason, Nick is also absolutely correct in criticizing the idea. A good idea will be able to stand up to critics. > 2) Is Python not an open-source project? Could not anybody contribute? To me > that means, the sky is the limit. "Open source" and "free software" and so on do not mean that anyone can make changes to the core. It does mean, however, that you're legally, morally, and ethically permitted to make your own version of CPython that includes the "else if" notation, play around with it, and demonstrate your idea in action. This is actually a great way to support an idea; you can find out exactly how much code would have to change to make this work, and start playing around with it. And then if the idea does end up being accepted, your patch can be incorporated into the main body of code. > 3) Well, that the features you mentioned really do what they promise can be > only be proved if they are implemented and used. So, we will see how they > fare. Same, btw., would be for "else if". All of your and mine great > thoughts are just that: thoughts, assumptions, arguments. Not facts. > Yep. And that's why it helps to spin up a patch. Unfortunately this is a change to the language grammar, which is not the easiest thing to tweak (if you were proposing a change to a pure Python module in the standard library, you could play with it much more easily); but if you write the patch yourself, you are directly answering (part of) Nick's objection about opportunity cost. Personally, I'm not in favour of adding another way of spelling elif. If every other language in the world spelled it "else if" and Python alone used "elif", then I'd support adding the alias, for the convenience of people working with multiple languages; but it's not that consistent (some languages use "elsif", "elseif", and various other forms). The readability argument is weak, and I don't see any reason to create two distinct ways of spelling the exact same thing. But hey! Play with the patch, have some fun with it. ChrisA From vgr255 at live.ca Sun Oct 25 21:28:16 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Sun, 25 Oct 2015 21:28:16 -0400 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: , , <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com>, , , , <56290D47.9030800@mail.de>, , <562B43DF.9030809@mail.de>, Message-ID: Here is yet another argument against this change -- indentation. Indentation is, as we all well know, one of the core aspects of Python's syntax. In every project I have worked on, and in PEP 8 ( https://www.python.org/dev/peps/pep-0008/#indentation ), 4-space indents are used. 'elif' is exactly 4 characters long, which means it lines up pretty well with other similar keywords such as 'else' or 'for' (with the space that follows it). In that regard, 'else if' would actually make code *harder* to read, not easier! In the same sense that 'def' (followed by a space) makes it very easy to spot the functions' names (if for some reason you don't have syntax highlighting ... ), 'elif' makes it easy to spot the beginning of another condition statement within a block. Cheers,-Emanuel Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Oct 26 00:07:51 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Oct 2015 21:07:51 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <20151023141857.0aec87e9@anarchist.wooz.org> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <20151023141857.0aec87e9@anarchist.wooz.org> Message-ID: It's not going to be codified in PEP 8. On Fri, Oct 23, 2015 at 11:18 AM, Barry Warsaw wrote: > On Oct 22, 2015, at 06:56 PM, Yury Selivanov wrote: > > >In principle, there is no reason why *both* of these groups > >of users can't use one tool and be happy. I propose to > >establish a convention in PEP 8, explaining that, while both > >literals are semantically equivalent, > > > >- r'..' strings *should* be used for regexps, > > > >- R'..' strings *should* be used for unstyled raw strings, > > > >and tools *should* treat them as such. > > > >All of this is merely about codifying the current status quo. > > What about all the regexps that are written without raw strings? Won't > PEP 8 > codification shame everyone into turn them into raw strings when they > really > don't need to? > > Cheers, > -Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Mon Oct 26 00:32:09 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 26 Oct 2015 04:32:09 +0000 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <562B43DF.9030809@mail.de> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: On 24/10/2015 09:39, Sven R. Kunze wrote: > On 23.10.2015 18:09, Nick Coghlan wrote: >> It's a big deal due to opportunity cost - any time that is spent on >> making a second spelling of "elif" possible is time not spent working >> on something that directly improves developer productivity (like >> format strings) or application correctness (like gradual typing >> support). > > 1) I may re-iterate. Such thinking is dangerous. It could mean > standstill. It prevents thinking and progress. That is python-ideas; not > python-controlling. > > 2) Is Python not an open-source project? Could not anybody contribute? > To me that means, the sky is the limit. > > 3) Well, that the features you mentioned really do what they promise can > be only be proved if they are implemented and used. So, we will see how > they fare. Same, btw., would be for "else if". All of your and mine > great thoughts are just that: thoughts, assumptions, arguments. Not facts. > I have a very strong feeling that you haven't got the faintest idea how the Python development process works, from the time that an idea gets put forward here, until the first full release with that shiny new feature that everybody has been wanting for since forever. To me the most important thing that needs sorting is the core workflow. Sure other things need doing, but to me "else if" doesn't even get on the bottom of the infinitely long list of Python jobs that need doing. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From ncoghlan at gmail.com Fri Oct 23 12:14:37 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2015 18:14:37 +0200 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> <858u6vpqpt.fsf@benfinney.id.au> Message-ID: On 22 October 2015 at 05:31, Nathaniel Smith wrote: > On Wed, Oct 21, 2015 at 7:44 PM, Ben Finney wrote: >> Yury Selivanov writes: >>> In the process, we had to make a decision on how to highlight raw >>> string literals -- r''. Many existing highlighters assume that all raw >>> strings are regexps, and highlight them as such, i.e. '\s' and '\n' >>> will be highlighted. >> >> That is evidently a simple mistake. Merely knowing that a token is a raw >> string does not justify the assumption that the string is a regular >> expression, or a filesystem entry name, or a line in a network protocol, >> or anything except plain text. > > This isn't necessarily true, just as a matter of like... epistemology. > For example, if hypothetically it turned out that 99% of raw strings > are in fact regular expressions, then knowing something is a raw > string would give you quite a bit of evidence that it's a regular > expression -- quite possibly enough to justify treating it as such for > something like code highlighting. > > I haven't actually gathered any data to find out how strong the > association between raw strings and regexen is, but it'd be pretty > easy for someone to do. (Parse a large corpus of python code to > extract all raw strings, randomly subsample 100 of them, review > manually to decide if each is a regex.) I'd expect Windows filesystem paths to win handily if we could scan all the Python code in the world, but they wouldn't show up in a scan of POSIX specific open source code. Cheers, Nick -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ian.g.kelly at gmail.com Mon Oct 26 01:58:09 2015 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 25 Oct 2015 23:58:09 -0600 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: On Sun, Oct 25, 2015 at 7:28 PM, Emanuel Barry wrote: > Here is yet another argument against this change -- indentation. Indentation > is, as we all well know, one of the core aspects of Python's syntax. In > every project I have worked on, and in PEP 8 ( > https://www.python.org/dev/peps/pep-0008/#indentation ), 4-space indents are > used. 'elif' is exactly 4 characters long, which means it lines up pretty > well with other similar keywords such as 'else' or 'for' (with the space > that follows it). In that regard, 'else if' would actually make code > *harder* to read, not easier! In the same sense that 'def' (followed by a > space) makes it very easy to spot the functions' names (if for some reason > you don't have syntax highlighting ... ), 'elif' makes it easy to spot the > beginning of another condition statement within a block. I'm not following. Can you provide an example of how this hurts indentation? It seems a stretch to suggest that "elif" lines up with "for" because you're including the space that follows "for", but you're excluding the space that follows "elif". I note that "else if" has the same length as "finally", if that matters. From tritium-list at sdamon.com Mon Oct 26 02:02:00 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 26 Oct 2015 02:02:00 -0400 Subject: [Python-ideas] Allowing def to assign to anything Message-ID: <562DC1D8.8060003@sdamon.com> In my code, I write a lot of dispatch dictionaries (for lack of a switch statement, but I will not hold my breath for that). In trying to make writing these dictionaries less annoying, I tend to use many lambdas. I can let you guess at what problems that has resulted in. Of course, the preferred way to write such dictionaries is by using a regular function, and adding that function to a dictionary. This isn't exactly a problem - it works, and works well, but it is annoying to write, and leaves artifacts of those functions in module scope. I propose a little bit of sugar to make this a little less annoying. If `def` is allowed to assign to anything (anything that is legal at the left hand side of an = in that scope), annoying artifacts go away. The syntax I propose should be backwards compatible. ``` dispatch = {} def dispatch['foo'](bar): return bar * bar ``` Does this make anything possible that is impossible now? No. But it does make the intent of the module author clear - the function is only ever intended to live inside that dict, or list, or other structure. This, to me, is less annoying to write, and is more readable. This obviously could be used outside of creating dispatch dictionaries, but that is the use case I would benefit from. From tritium-list at sdamon.com Mon Oct 26 02:19:43 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 26 Oct 2015 02:19:43 -0400 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: References: <562DC1D8.8060003@sdamon.com> Message-ID: <562DC5FF.1090508@sdamon.com> This too leaves artifacts in the module, which is one of the annoyances I am trying to eliminate. Though I do admit that it makes the intent of the author clear. It still feels less pythonic to me than allowing a statement that assigns to just assign to anything assignable. On 10/26/2015 02:15, Bruce Leban wrote: > Decorators easily support this: > > dispatch_table = {} > def dispatch(arg): > def inner(func): > dispatch_table[arg] = func > return func > return inner > > @dispatch('foo') > def dispatch_foo(bar): > pass -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Oct 26 02:20:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Oct 2015 17:20:37 +1100 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC1D8.8060003@sdamon.com> References: <562DC1D8.8060003@sdamon.com> Message-ID: On Mon, Oct 26, 2015 at 5:02 PM, Alexander Walters wrote: > If `def` is allowed to assign to anything (anything that is legal at the > left hand side of an = in that scope), annoying artifacts go away. The > syntax I propose should be backwards compatible. > > ``` > dispatch = {} > > def dispatch['foo'](bar): > return bar * bar > ``` > > Does this make anything possible that is impossible now? No. But it does > make the intent of the module author clear - the function is only ever > intended to live inside that dict, or list, or other structure. This, to > me, is less annoying to write, and is more readable. This obviously could > be used outside of creating dispatch dictionaries, but that is the use case > I would benefit from. I agree; the idea has been raised a few times, and I think it'd be helpful. It's probably not necessary to allow the _entire_ scope of "anything legal on the left of =", as that's pretty broad; even if the only form allowed were obj[key], it'd be useful. But for building a dispatch dictionary, you could simply decorate your functions with a capturer: dispatch = {} def cmd(func): dispatch[func.__name__] = func return func @cmd def foo(bar): return bar * bar You can even merge the decorator and the dict itself: class DispatchDict(dict): def __call__(self, func): self[func.__name__] = func return func dispatch = DispatchDict() @dispatch def foo(bar): return bar * bar This does require that your dict keys be legal identifiers (you can't do "def dispatch['!'](x):" as "@cmd def !(x)"), but for a lot of common cases, this does work. I've used this style for building argparse UIs and such, and it's a lot easier than most other options I've played with. ChrisA From abarnert at yahoo.com Mon Oct 26 02:19:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 25 Oct 2015 23:19:43 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC1D8.8060003@sdamon.com> References: <562DC1D8.8060003@sdamon.com> Message-ID: On Oct 25, 2015, at 23:02, Alexander Walters wrote: > > In my code, I write a lot of dispatch dictionaries (for lack of a switch statement, but I will not hold my breath for that). In trying to make writing these dictionaries less annoying, I tend to use many lambdas. I can let you guess at what problems that has resulted in. Of course, the preferred way to write such dictionaries is by using a regular function, and adding that function to a dictionary. This isn't exactly a problem - it works, and works well, but it is annoying to write, and leaves artifacts of those functions in module scope. I propose a little bit of sugar to make this a little less annoying. > > If `def` is allowed to assign to anything (anything that is legal at the left hand side of an = in that scope), annoying artifacts go away. The syntax I propose should be backwards compatible. Seems interesting. What's the name of the defined function? For an attribution like "spam.eggs" you'd probably want it to be "eggs", and I guess for "spam['eggs']" as well, but what about "spam['two words']" or "spam[2]"? I assume the qualname is just the name. Also, would this go through the descriptor mechanism if you def an attribution? From bruce at leban.us Mon Oct 26 02:15:44 2015 From: bruce at leban.us (Bruce Leban) Date: Sun, 25 Oct 2015 23:15:44 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC1D8.8060003@sdamon.com> References: <562DC1D8.8060003@sdamon.com> Message-ID: Decorators easily support this: dispatch_table = {} def dispatch(arg): def inner(func): dispatch_table[arg] = func return func return inner @dispatch('foo') def dispatch_foo(bar): pass --- Bruce Check out my new puzzle book: http://J.mp/ingToConclusions Get it free here: http://J.mp/ingToConclusionsFree (available on iOS) On Sun, Oct 25, 2015 at 11:02 PM, Alexander Walters wrote: > In my code, I write a lot of dispatch dictionaries (for lack of a switch > statement, but I will not hold my breath for that). In trying to make > writing these dictionaries less annoying, I tend to use many lambdas. I > can let you guess at what problems that has resulted in. Of course, the > preferred way to write such dictionaries is by using a regular function, > and adding that function to a dictionary. This isn't exactly a problem - > it works, and works well, but it is annoying to write, and leaves artifacts > of those functions in module scope. I propose a little bit of sugar to > make this a little less annoying. > > If `def` is allowed to assign to anything (anything that is legal at the > left hand side of an = in that scope), annoying artifacts go away. The > syntax I propose should be backwards compatible. > > ``` > dispatch = {} > > def dispatch['foo'](bar): > return bar * bar > ``` > > Does this make anything possible that is impossible now? No. But it does > make the intent of the module author clear - the function is only ever > intended to live inside that dict, or list, or other structure. This, to > me, is less annoying to write, and is more readable. This obviously could > be used outside of creating dispatch dictionaries, but that is the use case > I would benefit from. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Fri Oct 23 14:40:29 2015 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 23 Oct 2015 14:40:29 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562969B4.5080406@gmail.com> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> Message-ID: <562A7F1D.90108@nedbatchelder.com> On 10/22/15 6:56 PM, Yury Selivanov wrote: > In principle, there is no reason why *both* of these groups > of users can't use one tool and be happy. I propose to > establish a convention in PEP 8, explaining that, while both > literals are semantically equivalent, > > - r'..' strings *should* be used for regexps, > > - R'..' strings *should* be used for unstyled raw strings, > > and tools *should* treat them as such. > > All of this is merely about codifying the current status quo. But you are not codifying the status quo. The distinction you are proposing is one that you have invented. I have never used R"" strings. I think the best solution to the problem is to improve the highlighters, and luckily you have written one! To me, it is clear which of these strings is the regex: r"\d+" r"\dir" If the highlighters tried some heuristics, they could do a better job "being helpful" by making better guesses about the meaning of programs. I don't mind when highlighters make wrong guesses, as long as they don't ruin the entire rest of the file. But better guesses will be better. :) --Ned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Mon Oct 26 02:23:54 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 26 Oct 2015 02:23:54 -0400 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: References: <562DC1D8.8060003@sdamon.com> Message-ID: <562DC6FA.3060309@sdamon.com> I imagine in the case of assigning to a class or instance, the name would be the same (and binding to self would act the same) as if it were assigned in the traditional way. I do not propose that assigning to classes in this way be considered a good idea. as for when assigned to a data structure, my admittedly naive idea would be to set the name to '__none__' or some such. I am open to other ideas though. On 10/26/2015 02:19, Andrew Barnert wrote: > Seems interesting. > > What's the name of the defined function? For an attribution like "spam.eggs" you'd probably want it to be "eggs", and I guess for "spam['eggs']" as well, but what about "spam['two words']" or "spam[2]"? > > I assume the qualname is just the name. > > Also, would this go through the descriptor mechanism if you def an attribution? From tritium-list at sdamon.com Mon Oct 26 02:30:28 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 26 Oct 2015 02:30:28 -0400 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: References: <562DC1D8.8060003@sdamon.com> Message-ID: <562DC884.2080707@sdamon.com> On 10/26/2015 02:20, Chris Angelico wrote: > I agree; the idea has been raised a few times, and I think it'd be > helpful. It's probably not necessary to allow the _entire_ scope of > "anything legal on the left of =", as that's pretty broad; even if the > only form allowed were obj[key], it'd be useful. I agree, that perhaps the scope could be a little wide when put this way, but my instinct is that 'allow anything already legal for =' would be the path of least frustration when implementing this. I could be woefully wrong. I do at least wish to assign to object with __setitem__ defined. > > But for building a dispatch dictionary, you could simply decorate your > functions with a capturer: > > dispatch = {} > > def cmd(func): > dispatch[func.__name__] = func > return func > > @cmd > def foo(bar): > return bar * bar > > You can even merge the decorator and the dict itself: > > class DispatchDict(dict): > def __call__(self, func): > self[func.__name__] = func > return func > > dispatch = DispatchDict() > > @dispatch > def foo(bar): > return bar * bar > > This does require that your dict keys be legal identifiers (you can't > do "def dispatch['!'](x):" as "@cmd def !(x)"), but for a lot of > common cases, this does work. I've used this style for building > argparse UIs and such, and it's a lot easier than most other options > I've played with. This does, indeed, make life a bit easier in the here and now (and is similar to kitbashed techniques I already use). I am hoping to make that obsolete. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ - Alex From bruce at leban.us Mon Oct 26 02:49:18 2015 From: bruce at leban.us (Bruce Leban) Date: Sun, 25 Oct 2015 23:49:18 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC5FF.1090508@sdamon.com> References: <562DC1D8.8060003@sdamon.com> <562DC5FF.1090508@sdamon.com> Message-ID: On Sun, Oct 25, 2015 at 11:19 PM, Alexander Walters wrote: > This too leaves artifacts in the module, which is one of the annoyances I > am trying to eliminate. Though I do admit that it makes the intent of the > author clear. It still feels less pythonic to me than allowing a statement > that assigns to just assign to anything assignable. > Well, to some extent features that are already in the language feel more pythonic than features that aren't. Given that this is already easy to do, and the decorator solution is more powerful, the bar for modifying the language is higher. That aside, I like the fact that I can find these functions in the module. For one thing, it makes them testable in an obvious way (without going through the dispatch dictionary). Generally, module_name.function_name doesn't change during the course of program execution but there's no such expectation for dispatch_dict[function_name]. Furthermore, consider that def dispatch['foo'](bar): pass might look like a simple dict reference but is actually a call to __setitem__ which means this can have arbitrary side effects. Yes @decorators can have arbitrary side effects too. How do I find out the respective side effects? Perhaps: help(decorator) # obvious help(dispatch) # nope help(dispatch.__setitem__) # not obvious to me --- Bruce Check out my new puzzle book: http://J.mp/ingToConclusions Get it free here: http://J.mp/ingToConclusionsFree (available on iOS) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Oct 26 03:05:02 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 00:05:02 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC6FA.3060309@sdamon.com> References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> Message-ID: <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> On Oct 25, 2015, at 23:23, Alexander Walters wrote: > > I imagine in the case of assigning to a class or instance, the name would be the same (and binding to self would act the same) as if it were assigned in the traditional way. In what traditional way? When you def a function, its name is the name given in the def statement. If you later assign it to a member of an object, that doesn't change its name. So, that doesn't answer the question. As for the binding to self, that doesn't happen at assignment time, so it can't happen the same way as at assignment time. The binding happens later, each time the method is looked up. If an object doesn't have an attribute of the looked up name, but its type does, the type's attribute's __get__ method is called with the instance. So, there's nothing to do here at all; functions already have a __get__ method that returns a bound method (and if you assign a function to an instance rather than a type, it doesn't get bound). This is all explained pretty clearly in the descriptor HOWTO. But descriptors also work for setting, not just getting (which is how @property works), and that's what I was asking about. If I write "spam.eggs = 0", and type(spam) has a member named "eggs", its __set__ method will get called with the instance (spam) and the value (0). So, if I write "def spam.eggs(): pass", does it call the same descriptor method? > I do not propose that assigning to classes in this way be considered a good idea. OK, but unless you're actually proposing to not allow it, you still need to work out what it would do. > as for when assigned to a data structure, my admittedly naive idea would be to set the name to '__none__' or some such. I am open to other ideas though. Functions defined with lambda get the name "", so you'd probably want angle brackets here as well. But otherwise that seems reasonable, I guess. > >> On 10/26/2015 02:19, Andrew Barnert wrote: >> Seems interesting. >> >> What's the name of the defined function? For an attribution like "spam.eggs" you'd probably want it to be "eggs", and I guess for "spam['eggs']" as well, but what about "spam['two words']" or "spam[2]"? >> >> I assume the qualname is just the name. >> >> Also, would this go through the descriptor mechanism if you def an attribution? > From tritium-list at sdamon.com Mon Oct 26 03:09:13 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 26 Oct 2015 03:09:13 -0400 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> Message-ID: <562DD199.1040004@sdamon.com> to be clear, I propose the following to be equivalent (to clear up what I mean) ``` # classes: class Foo: def bar(self): pass # vs class Foo: pass def Foo.bar(self): pass # Instances: def foo(args): pass inst = Class() inst.foo = foo # vs inst = Class() def inst.foo(args): pass ``` does this make that clear? On 10/26/2015 03:05, Andrew Barnert wrote: > On Oct 25, 2015, at 23:23, Alexander Walters wrote: >> I imagine in the case of assigning to a class or instance, the name would be the same (and binding to self would act the same) as if it were assigned in the traditional way. > In what traditional way? When you def a function, its name is the name given in the def statement. If you later assign it to a member of an object, that doesn't change its name. So, that doesn't answer the question. > > As for the binding to self, that doesn't happen at assignment time, so it can't happen the same way as at assignment time. The binding happens later, each time the method is looked up. If an object doesn't have an attribute of the looked up name, but its type does, the type's attribute's __get__ method is called with the instance. So, there's nothing to do here at all; functions already have a __get__ method that returns a bound method (and if you assign a function to an instance rather than a type, it doesn't get bound). This is all explained pretty clearly in the descriptor HOWTO. > > But descriptors also work for setting, not just getting (which is how @property works), and that's what I was asking about. If I write "spam.eggs = 0", and type(spam) has a member named "eggs", its __set__ method will get called with the instance (spam) and the value (0). So, if I write "def spam.eggs(): pass", does it call the same descriptor method? > >> I do not propose that assigning to classes in this way be considered a good idea. > OK, but unless you're actually proposing to not allow it, you still need to work out what it would do. > >> as for when assigned to a data structure, my admittedly naive idea would be to set the name to '__none__' or some such. I am open to other ideas though. > Functions defined with lambda get the name "", so you'd probably want angle brackets here as well. But otherwise that seems reasonable, I guess. > >>> On 10/26/2015 02:19, Andrew Barnert wrote: >>> Seems interesting. >>> >>> What's the name of the defined function? For an attribution like "spam.eggs" you'd probably want it to be "eggs", and I guess for "spam['eggs']" as well, but what about "spam['two words']" or "spam[2]"? >>> >>> I assume the qualname is just the name. >>> >>> Also, would this go through the descriptor mechanism if you def an attribution? From ian.g.kelly at gmail.com Mon Oct 26 03:27:58 2015 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 26 Oct 2015 01:27:58 -0600 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: References: <562DC1D8.8060003@sdamon.com> Message-ID: > On Sun, Oct 25, 2015 at 11:02 PM, Alexander Walters > wrote: >> >> In my code, I write a lot of dispatch dictionaries (for lack of a switch >> statement, but I will not hold my breath for that). In trying to make >> writing these dictionaries less annoying, I tend to use many lambdas. I can >> let you guess at what problems that has resulted in. Of course, the >> preferred way to write such dictionaries is by using a regular function, and >> adding that function to a dictionary. This isn't exactly a problem - it >> works, and works well, but it is annoying to write, and leaves artifacts of >> those functions in module scope. I propose a little bit of sugar to make >> this a little less annoying. >> >> If `def` is allowed to assign to anything (anything that is legal at the >> left hand side of an = in that scope), annoying artifacts go away. The >> syntax I propose should be backwards compatible. >> >> ``` >> dispatch = {} >> >> def dispatch['foo'](bar): >> return bar * bar >> ``` What about: def foo(bar)[baz](x): return x This seems like it would complicate parsing as the parser can't be sure whether (bar) is a parameter list or an argument list until it reaches the following [baz]. From tritium-list at sdamon.com Mon Oct 26 03:23:31 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 26 Oct 2015 03:23:31 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562A7F1D.90108@nedbatchelder.com> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> Message-ID: <562DD4F3.5000901@sdamon.com> On 10/23/2015 14:40, Ned Batchelder wrote: > On 10/22/15 6:56 PM, Yury Selivanov wrote: >> In principle, there is no reason why *both* of these groups >> of users can't use one tool and be happy. I propose to >> establish a convention in PEP 8, explaining that, while both >> literals are semantically equivalent, >> >> - r'..' strings *should* be used for regexps, >> >> - R'..' strings *should* be used for unstyled raw strings, >> >> and tools *should* treat them as such. >> >> All of this is merely about codifying the current status quo. > But you are not codifying the status quo. The distinction you are > proposing is one that you have invented. I have never used R"" strings. > > I think the best solution to the problem is to improve the > highlighters, and luckily you have written one! To me, it is clear > which of these strings is the regex: > > r"\d+" > r"\dir" > > If the highlighters tried some heuristics, they could do a better job > "being helpful" by making better guesses about the meaning of > programs. I don't mind when highlighters make wrong guesses, as long > as they don't ruin the entire rest of the file. But better guesses > will be better. :) > > --Ned. > it should be noted that most regexes are also valid paths on NTFS. is r'\dir[a-zA-Z0-9]\\' a path or a regex? -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Oct 26 03:55:18 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 00:55:18 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DD199.1040004@sdamon.com> References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> <562DD199.1040004@sdamon.com> Message-ID: <00A91850-DD3B-4BA8-B78A-CA6EF7C16906@yahoo.com> On Oct 26, 2015, at 00:09, Alexander Walters wrote: > > to be clear, I propose the following to be equivalent (to clear up what I mean) Yes, that clears it up. This is what I thought you probably wanted. But I think it would be better to keep it simpler. Instead of this: > # classes: > > class Foo: > def bar(self): > pass ? your first example would be simpler if it were equivalent to this: class Foo: pass def _dummy(self): pass Foo.bar = _dummy (except of course not binding anything to "_dummy"). Notice that this is now exactly like your second example, with an instance, so we only need one rule instead of two. And in simple cases it does exactly what you want here, too. And in complex cases, e.g., involving "__class__" or no-argument "super" or names accessible in the class-definition scope, the implementation doesn't need to do any magic, while with your version, it would. (That magic might be convenient in some cases, but since you don't want to encourage this kind of use anyway, why make things more complicated just to make the discouraged use more convenient?) In fact, I would make this as simple as possible: Just change the "funcname" in the syntax from an identifier to a target. Then, any "def" statement whose works by defining a function, then assigning it to the target (which is legal iff the target is a legal assignment target). It's a perfectly normal assignment, following all the same __setitem__, __setattr__, descriptor, etc. rules as any other assignment. That does mean the name of the function ends up being "Foo.bar" or "foo.bar" or "foo['bar']", and at least for the first two could end up being confusing, and even more so for the qualname, so maybe you'd want to add one minor tweak: if the target is not an identifier, the name is just "" or "" or something. But I don't think you want to go any farther. (In particular, making any of the examples come out as "bar" is just extra complexity that only improves uses you don't want to encourage.) From tritium-list at sdamon.com Mon Oct 26 04:01:22 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 26 Oct 2015 04:01:22 -0400 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <00A91850-DD3B-4BA8-B78A-CA6EF7C16906@yahoo.com> References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> <562DD199.1040004@sdamon.com> <00A91850-DD3B-4BA8-B78A-CA6EF7C16906@yahoo.com> Message-ID: <562DDDD2.4090106@sdamon.com> On 10/26/2015 03:55, Andrew Barnert wrote: > ? your first example would be simpler if it were equivalent to this: > > class Foo: > pass > def _dummy(self): > pass > Foo.bar = _dummy > > (except of course not binding anything to "_dummy"). > > Notice that this is now exactly like your second example, with an instance, so we only need one rule instead of two. And in simple cases it does exactly what you want here, too. And in complex cases, e.g., involving "__class__" or no-argument "super" or names accessible in the class-definition scope, the implementation doesn't need to do any magic, while with your version, it would. (That magic might be convenient in some cases, but since you don't want to encourage this kind of use anyway, why make things more complicated just to make the discouraged use more convenient?) Agreed, that is a better idea. > In fact, I would make this as simple as possible: Just change the "funcname" in the syntax from an identifier to a target. Then, any "def" statement whose works by defining a function, then assigning it to the target (which is legal iff the target is a legal assignment target). It's a perfectly normal assignment, following all the same __setitem__, __setattr__, descriptor, etc. rules as any other assignment. > > That does mean the name of the function ends up being "Foo.bar" or "foo.bar" or "foo['bar']", and at least for the first two could end up being confusing, and even more so for the qualname, so maybe you'd want to add one minor tweak: if the target is not an identifier, the name is just "" or "" or something. But I don't think you want to go any farther. (In particular, making any of the examples come out as "bar" is just extra complexity that only improves uses you don't want to encourage.) > The more I think about it function.__name__ when assigned like this might be better served as `'' % (whatever_you_assigned_to,)` From p.f.moore at gmail.com Mon Oct 26 04:37:50 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Oct 2015 08:37:50 +0000 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: <562B43DF.9030809@mail.de> References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: On 24 October 2015 at 09:39, Sven R. Kunze wrote: > 2) Is Python not an open-source project? Could not anybody contribute? To me > that means, the sky is the limit. The problem is that many people contribute ideas (hence this list) but only a few contribute code, and even fewer contribute the commitment to maintain that code long-term. One of the roles needed on this list is for people to say that the *current* developers (i.e, those contributing code and maintenance) are unlikely to spend their time implementing and maintaining an idea proposed here. That doesn't mean the idea is rejected out of hand, it just means that to progress the idea the person proposing it is likely to have to go further, and offer code, and potentially ongoing maintenance if it's a change that can't be covered by "normal" maintenance (e.g., a inclusion of a new or specialist library). So ideas are welcome, but they need to be backed by an implementation, and "I'm not interested in implementing this idea for you" is an entirely legitimate (and not uncommon) response that people need to be prepared to accept. Paul From python at 2sn.net Mon Oct 26 04:57:29 2015 From: python at 2sn.net (Alexander Heger) Date: Mon, 26 Oct 2015 19:57:29 +1100 Subject: [Python-ideas] Allowing def to assign to anything Message-ID: how about class Dispatch(object): @staticmethod def foo(bar): return bar * bar def __getitem__(self, key): return self.__getattribute__(key) dispatch = Dispatch() >>> test.dispatch['foo'](2) 4 this leaves the module namespace less polluted. -Alexander On 26 October 2015 at 18:27, Ian Kelly wrote: >> On Sun, Oct 25, 2015 at 11:02 PM, Alexander Walters >> wrote: >>> >>> In my code, I write a lot of dispatch dictionaries (for lack of a switch >>> statement, but I will not hold my breath for that). In trying to make >>> writing these dictionaries less annoying, I tend to use many lambdas. I can >>> let you guess at what problems that has resulted in. Of course, the >>> preferred way to write such dictionaries is by using a regular function, and >>> adding that function to a dictionary. This isn't exactly a problem - it >>> works, and works well, but it is annoying to write, and leaves artifacts of >>> those functions in module scope. I propose a little bit of sugar to make >>> this a little less annoying. >>> >>> If `def` is allowed to assign to anything (anything that is legal at the >>> left hand side of an = in that scope), annoying artifacts go away. The >>> syntax I propose should be backwards compatible. >>> >>> ``` >>> dispatch = {} >>> >>> def dispatch['foo'](bar): >>> return bar * bar >>> ``` > > What about: > > def foo(bar)[baz](x): > return x > > This seems like it would complicate parsing as the parser can't be > sure whether (bar) is a parameter list or an argument list until it > reaches the following [baz]. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Mon Oct 26 05:08:18 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 02:08:18 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DDDD2.4090106@sdamon.com> References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> <562DD199.1040004@sdamon.com> <00A91850-DD3B-4BA8-B78A-CA6EF7C16906@yahoo.com> <562DDDD2.4090106@sdamon.com> Message-ID: On Oct 26, 2015, at 01:01, Alexander Walters wrote: > > >> On 10/26/2015 03:55, Andrew Barnert wrote: >> ? your first example would be simpler if it were equivalent to this: >> >> class Foo: >> pass >> def _dummy(self): >> pass >> Foo.bar = _dummy >> >> (except of course not binding anything to "_dummy"). >> >> Notice that this is now exactly like your second example, with an instance, so we only need one rule instead of two. And in simple cases it does exactly what you want here, too. And in complex cases, e.g., involving "__class__" or no-argument "super" or names accessible in the class-definition scope, the implementation doesn't need to do any magic, while with your version, it would. (That magic might be convenient in some cases, but since you don't want to encourage this kind of use anyway, why make things more complicated just to make the discouraged use more convenient?) > > Agreed, that is a better idea. > >> In fact, I would make this as simple as possible: Just change the "funcname" in the syntax from an identifier to a target. Then, any "def" statement whose works by defining a function, then assigning it to the target (which is legal iff the target is a legal assignment target). It's a perfectly normal assignment, following all the same __setitem__, __setattr__, descriptor, etc. rules as any other assignment. >> >> That does mean the name of the function ends up being "Foo.bar" or "foo.bar" or "foo['bar']", and at least for the first two could end up being confusing, and even more so for the qualname, so maybe you'd want to add one minor tweak: if the target is not an identifier, the name is just "" or "" or something. But I don't think you want to go any farther. (In particular, making any of the examples come out as "bar" is just extra complexity that only improves uses you don't want to encourage.) > The more I think about it function.__name__ when assigned like this might be better served as `'' % (whatever_you_assigned_to,)` Actually, now that I think about it, I think that would be up to each implementation?I don't think Python specifies "" as the name for lambdas, that's just what CPython does, so the same would be true here, right? But yeah, now that I see the idea, I definitely like '' a lot better than '' or 'foo.bar'. From ncoghlan at gmail.com Mon Oct 26 05:52:16 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Oct 2015 10:52:16 +0100 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: <5627D667.8020301@mail.de> <5627E6F7.50200@mrabarnett.plus.com> <56290D47.9030800@mail.de> <562B43DF.9030809@mail.de> Message-ID: On 26 October 2015 at 09:37, Paul Moore wrote: > So ideas are welcome, but they need to be backed by an implementation, > and "I'm not interested in implementing this idea for you" is an > entirely legitimate (and not uncommon) response that people need to be > prepared to accept. It's also the case that the number of ways we can make Python worse is uncountably infinite, while the number of ways we can make it easier to use without also making it harder to learn is vanishingly small by comparison. (Hence the controversy over concepts like gradual typing, which improves usability for larger projects and teams, but introduces a new concept that people are likely to eventually have to learn as they gain further experience with the language) This means that *both* python-ideas and python-dev serve primarily as filters when it comes to new features: python-ideas finds the ideas that *might* have potential, and helps refine them into a concrete pitch to python-dev (or an enhancement request on the issue tracker for simpler ideas that don't need full python-dev review), while python-dev is more focused on delivering a relatively prompt yes/no answer on whether or not an idea is acceptable in principle, and then refining the practical details if the answer is "yes". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ned at nedbatchelder.com Mon Oct 26 07:33:38 2015 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 26 Oct 2015 07:33:38 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562DD4F3.5000901@sdamon.com> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> Message-ID: <562E0F92.7050400@nedbatchelder.com> On 10/26/15 3:23 AM, Alexander Walters wrote: > > On 10/23/2015 14:40, Ned Batchelder wrote: >> On 10/22/15 6:56 PM, Yury Selivanov wrote: >>> In principle, there is no reason why *both* of these groups >>> of users can't use one tool and be happy. I propose to >>> establish a convention in PEP 8, explaining that, while both >>> literals are semantically equivalent, >>> >>> - r'..' strings *should* be used for regexps, >>> >>> - R'..' strings *should* be used for unstyled raw strings, >>> >>> and tools *should* treat them as such. >>> >>> All of this is merely about codifying the current status quo. >> But you are not codifying the status quo. The distinction you are >> proposing is one that you have invented. I have never used R"" strings. >> >> I think the best solution to the problem is to improve the >> highlighters, and luckily you have written one! To me, it is clear >> which of these strings is the regex: >> >> r"\d+" >> r"\dir" >> >> If the highlighters tried some heuristics, they could do a better job >> "being helpful" by making better guesses about the meaning of >> programs. I don't mind when highlighters make wrong guesses, as long >> as they don't ruin the entire rest of the file. But better guesses >> will be better. :) >> >> --Ned. >> > > it should be noted that most regexes are also valid paths on NTFS. is > r'\dir[a-zA-Z0-9]\\' a path or a regex? I understand developers' penchant for getting everything precisely right and accounting for the darkest of corners and the farthest reaches of obscure edge cases. But I'm talking about making a reasonable guess. If the string contains square brackets, especially paired brackets with hyphens inside, it's probably a regex. --Ned. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Oct 26 07:55:55 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 04:55:55 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <562E0F92.7050400@nedbatchelder.com> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> Message-ID: <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> On Oct 26, 2015, at 04:33, Ned Batchelder wrote: > > >> On 10/26/15 3:23 AM, Alexander Walters wrote: >> >>> On 10/23/2015 14:40, Ned Batchelder wrote: >>>> On 10/22/15 6:56 PM, Yury Selivanov wrote: >>>> In principle, there is no reason why *both* of these groups >>>> of users can't use one tool and be happy. I propose to >>>> establish a convention in PEP 8, explaining that, while both >>>> literals are semantically equivalent, >>>> >>>> - r'..' strings *should* be used for regexps, >>>> >>>> - R'..' strings *should* be used for unstyled raw strings, >>>> >>>> and tools *should* treat them as such. >>>> >>>> All of this is merely about codifying the current status quo. >>> But you are not codifying the status quo. The distinction you are proposing is one that you have invented. I have never used R"" strings. >>> >>> I think the best solution to the problem is to improve the highlighters, and luckily you have written one! To me, it is clear which of these strings is the regex: >>> >>> r"\d+" >>> r"\dir" >>> >>> If the highlighters tried some heuristics, they could do a better job "being helpful" by making better guesses about the meaning of programs. I don't mind when highlighters make wrong guesses, as long as they don't ruin the entire rest of the file. But better guesses will be better. :) >>> >>> --Ned. >> >> it should be noted that most regexes are also valid paths on NTFS. is r'\dir[a-zA-Z0-9]\\' a path or a regex? > I understand developers' penchant for getting everything precisely right and accounting for the darkest of corners and the farthest reaches of obscure edge cases. But I'm talking about making a reasonable guess. If the string contains square brackets, especially paired brackets with hyphens inside, it's probably a regex. From working on music tagging software, I can tell you that an awful lot of users have mp3s with square brackets, hyphens, and other such things in their filenames, so if your software makes any assumptions about what filenames look like, their libraries will break your software. And to verify that this isn't some weird artifact of the way people used to name files on piracy networks back when people traded individual songs, I went to The Pirate Bay and checked the most popular current download in any category, and its first file is named: [ www.CpasBien.pw ] Tomorrowland.2015.TRUEFRENCH.BDRip.VxiD-EXTREME.avi So, I don't think you can assume that paired square brackets or hyphens mean something is not a Windows pathname. Of course with a wide enough corpus of filenames people have to deal with, you could come up with a better heuristic. (Not many regexes have character classes that are dotted domain names, or match a standard language code followed by "-sub", or most of the other examples I see from a quick scan.) But just guessing based on what you guess filenames are like without looking around is not going to get you that far. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Oct 26 09:45:39 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Oct 2015 00:45:39 +1100 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC1D8.8060003@sdamon.com> References: <562DC1D8.8060003@sdamon.com> Message-ID: <20151026134537.GX3813@ando.pearwood.info> On Mon, Oct 26, 2015 at 02:02:00AM -0400, Alexander Walters wrote: > In my code, I write a lot of dispatch dictionaries (for lack of a switch > statement, but I will not hold my breath for that). In trying to make > writing these dictionaries less annoying, I tend to use many lambdas. I > can let you guess at what problems that has resulted in. Of course, the > preferred way to write such dictionaries is by using a regular function, > and adding that function to a dictionary. This isn't exactly a problem > - it works, and works well, but it is annoying to write, and leaves > artifacts of those functions in module scope. I propose a little bit of > sugar to make this a little less annoying. In this case, leaving "artifacts" in the module scope is a feature. If your function is simple enough to express in a simple expression, then a lambda may be the right solution. But if it requires a full block, then chances are that it's too complex for it to be obviously correct, which means you should test it. Giving the function a name and module scope supports testing. But if you really want to get rid of it: del the_function after adding it to the dispatch table. Or, stick them in their own namespace. For a package, that might mean moving the dispatch table and its associated functions into its own module. Or, put them in a class: class Switch: def cheese(x): ... def spam(x): ... def eggs(x): ... dispatch = {} dispatch.update(Switch.__dict__) result = dispatch[key](arg) Or one could write a class decorator (or a metaclass) to post-process the class and returns whatever you like. If I were doing this a lot, I would invest the time in building a nice switch construct, before looking for new syntax. > If `def` is allowed to assign to anything (anything that is legal at the > left hand side of an = in that scope), annoying artifacts go away. The > syntax I propose should be backwards compatible. Assign to *anything*? a, b, c = 1, 2, 3 def a, b, c(x, y): ... I don't see that, at least, working. > ``` > dispatch = {} > > def dispatch['foo'](bar): > return bar * bar > ``` > > Does this make anything possible that is impossible now? No. But it > does make the intent of the module author clear - the function is only > ever intended to live inside that dict, or list, or other structure. It's not clear to me. To me, it looks like you've badly mistyped a function annotation. If you hadn't explained what you wanted, I wouldn't have a clue what "def dispatch['foo'](bar)" meant. -- Steve From vgr255 at live.ca Mon Oct 26 09:57:01 2015 From: vgr255 at live.ca (Emanuel Barry) Date: Mon, 26 Oct 2015 09:57:01 -0400 Subject: [Python-ideas] "else if" as equivalent for "elif" In-Reply-To: References: , <5627D667.8020301@mail.de>, <5627E6F7.50200@mrabarnett.plus.com>, , , , <56290D47.9030800@mail.de>, , <562B43DF.9030809@mail.de>, , , Message-ID: > From: ian.g.kelly at gmail.com > Date: Sun, 25 Oct 2015 23:58:09 -0600 > To: python-ideas at python.org > Subject: Re: [Python-ideas] "else if" as equivalent for "elif" > > On Sun, Oct 25, 2015 at 7:28 PM, Emanuel Barry wrote: > > Here is yet another argument against this change -- indentation. Indentation > > is, as we all well know, one of the core aspects of Python's syntax. In > > every project I have worked on, and in PEP 8 ( > > https://www.python.org/dev/peps/pep-0008/#indentation ), 4-space indents are > > used. 'elif' is exactly 4 characters long, which means it lines up pretty > > well with other similar keywords such as 'else' or 'for' (with the space > > that follows it). In that regard, 'else if' would actually make code > > *harder* to read, not easier! In the same sense that 'def' (followed by a > > space) makes it very easy to spot the functions' names (if for some reason > > you don't have syntax highlighting ... ), 'elif' makes it easy to spot the > > beginning of another condition statement within a block. > > I'm not following. Can you provide an example of how this hurts > indentation? It seems a stretch to suggest that "elif" lines up with > "for" because you're including the space that follows "for", but > you're excluding the space that follows "elif". > > I note that "else if" has the same length as "finally", if that matters. I agree it might sound a bit inconsistent that I take into account the space following 'for' but not the one following 'elif' - my point is that a human reader can easily look at the first characters before the next indentation level kicks in. This might not be a good example though now that I think of it, as a human reader would see that the line is not just 'else' (which was pretty much my original point). I recognize that you're right though, and I take back my words. Cheers,-Emanuel Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Oct 26 10:30:12 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 26 Oct 2015 09:30:12 -0500 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC1D8.8060003@sdamon.com> References: <562DC1D8.8060003@sdamon.com> Message-ID: <6CF8150C-9696-46C0-B72D-6D9ED0095E78@gmail.com> You can just do: dispatch = {} def dispatcher(f): dispatch[f.__name__] = f @dispatcher def foo(bar): return bar * bar I really think that might be weird and kind of jarring to read... On October 26, 2015 1:02:00 AM CDT, Alexander Walters wrote: >In my code, I write a lot of dispatch dictionaries (for lack of a >switch >statement, but I will not hold my breath for that). In trying to make >writing these dictionaries less annoying, I tend to use many lambdas. >I >can let you guess at what problems that has resulted in. Of course, >the >preferred way to write such dictionaries is by using a regular >function, >and adding that function to a dictionary. This isn't exactly a problem > >- it works, and works well, but it is annoying to write, and leaves >artifacts of those functions in module scope. I propose a little bit >of >sugar to make this a little less annoying. > >If `def` is allowed to assign to anything (anything that is legal at >the >left hand side of an = in that scope), annoying artifacts go away. The > >syntax I propose should be backwards compatible. > >``` >dispatch = {} > >def dispatch['foo'](bar): > return bar * bar >``` > >Does this make anything possible that is impossible now? No. But it >does make the intent of the module author clear - the function is only >ever intended to live inside that dict, or list, or other structure. >This, to me, is less annoying to write, and is more readable. This >obviously could be used outside of creating dispatch dictionaries, but >that is the use case I would benefit from. >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Mon Oct 26 11:17:43 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 26 Oct 2015 11:17:43 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <20151023014208.GQ3813@ando.pearwood.info> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <20151023014208.GQ3813@ando.pearwood.info> Message-ID: <562E4417.4090709@gmail.com> On 2015-10-22 9:42 PM, Steven D'Aprano wrote: > If you want my advice, it would be to either just pick whichever style > of formatting you like, and people will either like your code > highlighter and use it, or they won't, or you should consider a plug-in > architecture where your users can choose the string-highlighter style > they want, and even extend it to other DSLs. Yeah, I finally figured out a way to make the highlighter easily configurable, so that users can turn on/off highlighting of regexes, formatting literals, python2 support etc. This discussion was actually quite useful to me. Thanks to everybody for feedback! Yury From abarnert at yahoo.com Mon Oct 26 15:30:05 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 12:30:05 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <20151026134537.GX3813@ando.pearwood.info> References: <562DC1D8.8060003@sdamon.com> <20151026134537.GX3813@ando.pearwood.info> Message-ID: On Oct 26, 2015, at 06:45, Steven D'Aprano wrote: > >> On Mon, Oct 26, 2015 at 02:02:00AM -0400, Alexander Walters wrote: >> >> In my code, I write a lot of dispatch dictionaries (for lack of a switch >> statement, but I will not hold my breath for that). In trying to make >> writing these dictionaries less annoying, I tend to use many lambdas. I >> can let you guess at what problems that has resulted in. Of course, the >> preferred way to write such dictionaries is by using a regular function, >> and adding that function to a dictionary. This isn't exactly a problem >> - it works, and works well, but it is annoying to write, and leaves >> artifacts of those functions in module scope. I propose a little bit of >> sugar to make this a little less annoying. > > In this case, leaving "artifacts" in the module scope is a feature. If > your function is simple enough to express in a simple expression, then a > lambda may be the right solution. But if it requires a full block, then > chances are that it's too complex for it to be obviously correct, which > means you should test it. There is a bit of a boundary here: some functions are a single statement, which can't be written as a lambda, but are just as trivial as functions that can be. And functions that require two statements aren't that much more complex. And meanwhile, there are plenty of people who twist things into knots to fit things in expressions that don't belong as expressions just so they can use lambda, and this proposal could give their colleagues/teacher/conscience a better way to say "that should be a def" and answer the objections about "but that would mean 3 lines of boilerplate for a 2-line function". Of course without a realistic example instead of just the empty toy definitions in the original proposal, it's hard to see if there really would be such benefits here, or if it's just a theoretical possibility that would almost never arise, so it's definitely worth you pointing out this issue. > Giving the function a name and module scope > supports testing. It's not that hard to write unit tests that call dispatcher['spam'] instead of calling spam. Your unit test framework may not do this out of the box, but if you're going to be testing lots of functions like this, that's just something you have to add support for once. > But if you really want to get rid of it: > > del the_function > > after adding it to the dispatch table. Or, if you're worried about people calling them accidentally, just have the decorator return None so that's an error. Or, if you're worried about them showing up in tab completion, import *, etc., prefix them with _ and/or leave them out of __all__. The only time "polluting the namespace" is literally a concern is if you might have something else with the same name and you don't want one of them to erase the other; usually it's one of these other things you're really concerned with. > Or, stick them in their own namespace. For a package, that might mean > moving the dispatch table and its associated functions into its own > module. Or, put them in a class: Or just local to a function: def make_dispatch_table(): def spam(x): pass def eggs(x): pass def cheese(x): pass return {'spam': spam, 'eggs': eggs, 'cheese': cheese} Or, if you really want, you could even replace the last line with: return {k: v for (k, v) in locals().items() if callable(v)} Anyway, it's worth noting that your submodule idea and the local function idea share something in common: because there is a scope the functions are defined in, and they still live in that scope, they can call each other. The OP's suggestion, and your del suggestion, make that impossible. Also, inspect, or manual inspection by hacking away at the REPL, will work a lot more easily. I still think this proposal (suitably worked out) isn't a bad idea, but I agree that you've given good reasons why it isn't necessary for the stated use case, and why we need better and/or more complete examples to evaluate it. >> If `def` is allowed to assign to anything (anything that is legal at the >> left hand side of an = in that scope), annoying artifacts go away. The >> syntax I propose should be backwards compatible. > > Assign to *anything*? This is part of why I was trying to get him to refine it into a complete specification instead of a vague idea. The start of the answer here is pretty obvious: "def" takes an assignment target?not a target list, or another assignment, even though of course both of those are legal on the left side of an "=". (And further extensions to iterable unpacking, or even full pattern-matching assignment, wouldn't change what "target" means.) But that still isn't a complete answer. There are things that are allowed as targets by the grammar, but are still syntax errors ("*spam = 2") in an assignment, and I'm pretty sure the list of such things would be slightly different for def statements. (For example, a target can be a parenthesized target list, or a slicing. For a single assignment, that's not a syntax error, although of course it's a type error at runtime if the value isn't iterable; for a function definition, that should probably be a syntax error.) So someone still needs to write out a semantic specification like the one in the docs for assignment statements. But at least starting with "an assignment target" instead of "anything to the left of '='" is a start. It's as good as writing out the grammar, and a lot easier for most people to understand, and I think it gets the intuitive idea across. From abarnert at yahoo.com Mon Oct 26 15:38:42 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 12:38:42 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> Message-ID: On Oct 26, 2015, at 04:55, Andrew Barnert via Python-ideas wrote: > >> On Oct 26, 2015, at 04:33, Ned Batchelder wrote: >> >> >>> On 10/26/15 3:23 AM, Alexander Walters wrote: >>> >>>> On 10/23/2015 14:40, Ned Batchelder wrote: >>>>> On 10/22/15 6:56 PM, Yury Selivanov wrote: >>>>> In principle, there is no reason why *both* of these groups >>>>> of users can't use one tool and be happy. I propose to >>>>> establish a convention in PEP 8, explaining that, while both >>>>> literals are semantically equivalent, >>>>> >>>>> - r'..' strings *should* be used for regexps, >>>>> >>>>> - R'..' strings *should* be used for unstyled raw strings, >>>>> >>>>> and tools *should* treat them as such. >>>>> >>>>> All of this is merely about codifying the current status quo. >>>> But you are not codifying the status quo. The distinction you are proposing is one that you have invented. I have never used R"" strings. >>>> >>>> I think the best solution to the problem is to improve the highlighters, and luckily you have written one! To me, it is clear which of these strings is the regex: >>>> >>>> r"\d+" >>>> r"\dir" >>>> >>>> If the highlighters tried some heuristics, they could do a better job "being helpful" by making better guesses about the meaning of programs. I don't mind when highlighters make wrong guesses, as long as they don't ruin the entire rest of the file. But better guesses will be better. :) >>>> >>>> --Ned. >>> >>> it should be noted that most regexes are also valid paths on NTFS. is r'\dir[a-zA-Z0-9]\\' a path or a regex? >> I understand developers' penchant for getting everything precisely right and accounting for the darkest of corners and the farthest reaches of obscure edge cases. But I'm talking about making a reasonable guess. If the string contains square brackets, especially paired brackets with hyphens inside, it's probably a regex. > > From working on music tagging software, I can tell you that an awful lot of users have mp3s with square brackets, hyphens, and other such things in their filenames, so if your software makes any assumptions about what filenames look like, their libraries will break your software. > > And to verify that this isn't some weird artifact of the way people used to name files on piracy networks back when people traded individual songs, I went to The Pirate Bay and checked the most popular current download in any category, and its first file is named: > > [ www.CpasBien.pw ] Tomorrowland.2015.TRUEFRENCH.BDRip.VxiD-EXTREME.avi > > So, I don't think you can assume that paired square brackets or hyphens mean something is not a Windows pathname. > > Of course with a wide enough corpus of filenames people have to deal with, you could come up with a better heuristic. (Not many regexes have character classes that are dotted domain names, or match a standard language code followed by "-sub", or most of the other examples I see from a quick scan.) But just guessing based on what you guess filenames are like without looking around is not going to get you that far. Just for fun: is there a Python regex that matches all valid Python regexes? Obviously there's no actual regular expression that matches all regular expressions (you can't handle matched brackets without recursion or some other extension). I think a Python regex to match all actual regular expressions should be pretty easy. But I'm not sure about a Python regex to match all Python regexes. (Although I'll bet if it's possible for perl, someone has written a CPAN module. Probably without "verbose-mode" whitespace or comments.) That still wouldn't solve the problem of the many things that are valid regexes and also valid Windows paths (not to mention valid doc strings with embedded code examples, because that includes any possible string?), or detecting things that are obviously intended to be regexes even though they're invalid, etc., so it's probably not very useful for this heuristic anyway. Hence the "just for fun"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Mon Oct 26 17:44:12 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 27 Oct 2015 08:44:12 +1100 Subject: [Python-ideas] PEP 8: raw strings & regular expressions References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> Message-ID: <85pp01mhjn.fsf@benfinney.id.au> Andrew Barnert via Python-ideas writes: > Just for fun: is there a Python regex that matches all valid Python > regexes? Yes: ?.*? matches all valid Python regexes. > Obviously there's no actual regular expression that matches all > regular expressions (you can't handle matched brackets without > recursion or some other extension). You seem to be seeking something else: a pattern that matches all valid regex patterns, *and* will never match any string that is not a valid regex pattern. The latter is rather more difficult. -- \ ?When I was born I was so surprised I couldn't talk for a year | `\ and a half.? ?Gracie Allen | _o__) | Ben Finney From mertz at gnosis.cx Mon Oct 26 17:53:48 2015 From: mertz at gnosis.cx (David Mertz) Date: Mon, 26 Oct 2015 14:53:48 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <85pp01mhjn.fsf@benfinney.id.au> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> <85pp01mhjn.fsf@benfinney.id.au> Message-ID: Obviously there can't be a regex to exclude everything that isn't a regex. Parentheses can nest to unlimited depths, so you need a formal grammar. But virtually everything that is a Windows path is also formally grammatical regex too (as are many things with no plausible likely intention as such) On Oct 26, 2015 2:44 PM, "Ben Finney" wrote: > Andrew Barnert via Python-ideas > writes: > > > Just for fun: is there a Python regex that matches all valid Python > > regexes? > > Yes: ?.*? matches all valid Python regexes. > > > Obviously there's no actual regular expression that matches all > > regular expressions (you can't handle matched brackets without > > recursion or some other extension). > > You seem to be seeking something else: a pattern that matches all valid > regex patterns, *and* will never match any string that is not a valid > regex pattern. The latter is rather more difficult. > > -- > \ ?When I was born I was so surprised I couldn't talk for a year | > `\ and a half.? ?Gracie Allen | > _o__) | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Oct 26 18:23:39 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 15:23:39 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> <85pp01mhjn.fsf@benfinney.id.au> Message-ID: On Oct 26, 2015, at 14:53, David Mertz wrote: > Obviously there can't be a regex to exclude everything that isn't a regex. Parentheses can nest to unlimited depths, so you need a formal grammar. > As I said: >> Obviously there's no actual regular expression that matches all regular expressions (you can't handle matched brackets without recursion or some other extension) But you can do it trivially with Perl, or with the Regex module for Python, e.g., just by sticking a "(?1)" inside a pair of escaped parens plus a negative lookahead or nongreedy repetition. I'm not sure exactly how powerful Python (re module) regexes are (if I want to match something that isn't a regular language, I tend to reach for or build a dedicated parser rather than try to stretch re), but I know they're somewhere between actual regular expressions and perl regexes. > But virtually everything that is a Windows path is also formally grammatical regex too (as are many things with no plausible likely intention as such) > That's not true. You can, for example, have unclosed brackets or parens in a Windows path. And if you're wondering why anyone would do that, consider MP3 files auto-named based on their ID3v1/FreeDB metadata, which truncates fields at 29 or 30 bytes. Anyway, as I said in the same message, it wouldn't be a useful heuristic because there's so much overlap, but you don't need to exaggerate that to make the same point. >> On Oct 26, 2015 2:44 PM, "Ben Finney" wrote: >> Andrew Barnert via Python-ideas >> writes: >> >> > Just for fun: is there a Python regex that matches all valid Python >> > regexes? >> >> Yes: ?.*? matches all valid Python regexes. >> >> > Obviously there's no actual regular expression that matches all >> > regular expressions (you can't handle matched brackets without >> > recursion or some other extension). >> >> You seem to be seeking something else: a pattern that matches all valid >> regex patterns, *and* will never match any string that is not a valid >> regex pattern. The latter is rather more difficult. >> >> -- >> \ ?When I was born I was so surprised I couldn't talk for a year | >> `\ and a half.? ?Gracie Allen | >> _o__) | >> Ben Finney >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon Oct 26 18:31:37 2015 From: mertz at gnosis.cx (David Mertz) Date: Mon, 26 Oct 2015 15:31:37 -0700 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> <85pp01mhjn.fsf@benfinney.id.au> Message-ID: I think we're down to quibbling over the meaning of "virtually" here. I recognize it's "not all" and you recognize it's "most" windows paths are grammatically regexen. So is it 80%, 95%, 99.9%? And do we mean "paths found in the wild" or "paths as systematically enumerated" from possibility space? On Oct 26, 2015 3:23 PM, "Andrew Barnert" wrote: > On Oct 26, 2015, at 14:53, David Mertz wrote: > > Obviously there can't be a regex to exclude everything that isn't a regex. > Parentheses can nest to unlimited depths, so you need a formal grammar. > > As I said: > > Obviously there's no actual regular expression that matches all regular >> expressions (you can't handle matched brackets without recursion or some >> other extension) > > > But you can do it trivially with Perl, or with the Regex module for > Python, e.g., just by sticking a "(?1)" inside a pair of escaped parens > plus a negative lookahead or nongreedy repetition. I'm not sure exactly how > powerful Python (re module) regexes are (if I want to match something that > isn't a regular language, I tend to reach for or build a dedicated parser > rather than try to stretch re), but I know they're somewhere between actual > regular expressions and perl regexes. > > But virtually everything that is a Windows path is also formally > grammatical regex too (as are many things with no plausible likely > intention as such) > > That's not true. You can, for example, have unclosed brackets or parens in > a Windows path. And if you're wondering why anyone would do that, consider > MP3 files auto-named based on their ID3v1/FreeDB metadata, which truncates > fields at 29 or 30 bytes. > > Anyway, as I said in the same message, it wouldn't be a useful heuristic > because there's so much overlap, but you don't need to exaggerate that to > make the same point. > > On Oct 26, 2015 2:44 PM, "Ben Finney" wrote: > >> Andrew Barnert via Python-ideas >> writes: >> >> > Just for fun: is there a Python regex that matches all valid Python >> > regexes? >> >> Yes: ?.*? matches all valid Python regexes. >> >> > Obviously there's no actual regular expression that matches all >> > regular expressions (you can't handle matched brackets without >> > recursion or some other extension). >> >> You seem to be seeking something else: a pattern that matches all valid >> regex patterns, *and* will never match any string that is not a valid >> regex pattern. The latter is rather more difficult. >> >> -- >> \ ?When I was born I was so surprised I couldn't talk for a year | >> `\ and a half.? ?Gracie Allen | >> _o__) | >> Ben Finney >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Oct 26 21:43:23 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Oct 2015 21:43:23 -0400 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> Message-ID: On 10/26/2015 3:38 PM, Andrew Barnert via Python-ideas wrote: > Just for fun: is there a Python regex that matches all valid Python > regexes? Could you take this discussion to python-list. The PEP8 proposal has been rejected. -- Terry Jan Reedy From tjreedy at udel.edu Mon Oct 26 22:04:04 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Oct 2015 22:04:04 -0400 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <20151026134537.GX3813@ando.pearwood.info> References: <562DC1D8.8060003@sdamon.com> <20151026134537.GX3813@ando.pearwood.info> Message-ID: On 10/26/2015 9:45 AM, Steven D'Aprano wrote: > On Mon, Oct 26, 2015 at 02:02:00AM -0400, Alexander Walters wrote: > >> In my code, I write a lot of dispatch dictionaries (for lack of a switch >> statement, but I will not hold my breath for that). In trying to make >> writing these dictionaries less annoying, I tend to use many lambdas. I >> can let you guess at what problems that has resulted in. Of course, the >> preferred way to write such dictionaries is by using a regular function, >> and adding that function to a dictionary. This isn't exactly a problem >> - it works, and works well, but it is annoying to write, and leaves >> artifacts of those functions in module scope. I propose a little bit of >> sugar to make this a little less annoying. > > In this case, leaving "artifacts" in the module scope is a feature. If > your function is simple enough to express in a simple expression, then a > lambda may be the right solution. But if it requires a full block, then > chances are that it's too complex for it to be obviously correct, which > means you should test it. Giving the function a name and module scope > supports testing. > > But if you really want to get rid of it: > > del the_function > > after adding it to the dispatch table. > > Or, stick them in their own namespace. For a package, that might mean > moving the dispatch table and its associated functions into its own > module. Or, put them in a class: > > class Switch: > def cheese(x): > ... > def spam(x): > ... > def eggs(x): > ... > > dispatch = {} > dispatch.update(Switch.__dict__) dispatch = dict(Switch.__dict__) Except for the extraneous dunder names in __dict__, I like this. For the fastidious, dispatch = {k:v for k, v in Switch.__dict__.items() if k[0] != '_'} This could be wrapped in a function that would take either a module (globals()) or class (.__dict__) as the namespace source. -- Terry Jan Reedy From rosuav at gmail.com Mon Oct 26 22:20:20 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Oct 2015 13:20:20 +1100 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <562DC884.2080707@sdamon.com> References: <562DC1D8.8060003@sdamon.com> <562DC884.2080707@sdamon.com> Message-ID: On Mon, Oct 26, 2015 at 5:30 PM, Alexander Walters wrote: > > > On 10/26/2015 02:20, Chris Angelico wrote: >> >> I agree; the idea has been raised a few times, and I think it'd be >> helpful. It's probably not necessary to allow the _entire_ scope of >> "anything legal on the left of =", as that's pretty broad; even if the >> only form allowed were obj[key], it'd be useful. > > > I agree, that perhaps the scope could be a little wide when put this way, > but my instinct is that 'allow anything already legal for =' would be the > path of least frustration when implementing this. I could be woefully > wrong. I do at least wish to assign to object with __setitem__ defined. If you start by allowing only obj[key] (aka obj.__setitem__), and maybe with only string literals or simple names for the key, it would be a strict subset of Python syntax. That means that a future proposal can easily expand on that (*un*like, for instance, the way "{obj[key]}".format(obj=obj) operates), while providing only the one or two most common usages: def dispatch["foo"](x, y): pass name = "bar" def dispatch[name](x, y): pass >> But for building a dispatch dictionary, you could simply decorate your >> functions with a capturer: >> > This does, indeed, make life a bit easier in the here and now (and is > similar to kitbashed techniques I already use). I am hoping to make that > obsolete. Sure. You'll need some strong use-cases that justify _not_ using this technique, though; since there's an existing way to do this, the bar for new syntax is "show why the current way isn't good enough". If you're worried about junk in your globals, what you could do is: dispatch = {} def cleanup(): self = cleanup # we're going to delete ourselves, so snapshot to local for n,v in globals(): if v is self: del globals()[n] def cmd(func): dispatch[func.__name__] = func return cleanup @cmd def foo(): pass @cmd def bar(): pass cleanup() # This function will self-destruct in three seconds. Two... one... After all, a decorator can return anything it likes! ChrisA From rosuav at gmail.com Mon Oct 26 22:48:36 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Oct 2015 13:48:36 +1100 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> Message-ID: On Mon, Oct 26, 2015 at 6:05 PM, Andrew Barnert via Python-ideas wrote: > On Oct 25, 2015, at 23:23, Alexander Walters wrote: >> >> I imagine in the case of assigning to a class or instance, the name would be the same (and binding to self would act the same) as if it were assigned in the traditional way. > > In what traditional way? When you def a function, its name is the name given in the def statement. If you later assign it to a member of an object, that doesn't change its name. So, that doesn't answer the question. > dispatch = {} def dispatch['foo'](): pass assert dispatch['foo'].__name__ == "dispatch['foo']" I have no problem with this, conceptually. Syntactically, it would be defined as f"dispatch[{key!r}]" if the key is allowed to be an arbitrary expression. ChrisA From rosuav at gmail.com Mon Oct 26 22:26:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Oct 2015 13:26:23 +1100 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: <85pp01mhjn.fsf@benfinney.id.au> References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> <85pp01mhjn.fsf@benfinney.id.au> Message-ID: On Tue, Oct 27, 2015 at 8:44 AM, Ben Finney wrote: > You seem to be seeking something else: a pattern that matches all valid > regex patterns, *and* will never match any string that is not a valid > regex pattern. The latter is rather more difficult. "Rather more difficult" may be an understatement. A regex can contain grouping parentheses which can arbitrarily nest, and matching that with a regex is, AIUI, fundamentally impossible. So I don't think it's possible to have a regex that validates a regex. Fortunately, it's easy to write a function that validates a regex. def is_regex(s): try: re.compile(s) except re.error: return False return True ChrisA From abarnert at yahoo.com Mon Oct 26 23:48:11 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 26 Oct 2015 20:48:11 -0700 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> Message-ID: On Oct 26, 2015, at 19:48, Chris Angelico wrote: > > On Mon, Oct 26, 2015 at 6:05 PM, Andrew Barnert via Python-ideas > wrote: >> On Oct 25, 2015, at 23:23, Alexander Walters wrote: >>> >>> I imagine in the case of assigning to a class or instance, the name would be the same (and binding to self would act the same) as if it were assigned in the traditional way. >> >> In what traditional way? When you def a function, its name is the name given in the def statement. If you later assign it to a member of an object, that doesn't change its name. So, that doesn't answer the question. >> > > dispatch = {} > def dispatch['foo'](): pass > assert dispatch['foo'].__name__ == "dispatch['foo']" > > I have no problem with this, conceptually. The paragraph you're replying to was about the naming for "def foo.bar():", which Alexander had suggested would work differently from "def foo['bar']():" (which he later retracted). So I'm not sure why you're proposing a different alternative for "def foo['bar']():" in response to these paragraphs. At any rate, later in the thread, Alexander suggested something like "", which I like better. The angle brackets and "anon" tell you that it's not guaranteed to be a REPL-friendly name?which would be possible in this case, but not in general. Plus, the part after the "anon" is just the text of the target node, so it's dead simple. (And if you're worried that this means the function's repr ends up with nested angle brackets?well, it does, but the same is true for lambda functions, and the hidden functions on comprehensions.) > Syntactically, it would be > defined as f"dispatch[{key!r}]" if the key is allowed to be an > arbitrary expression. But surely dispatch is also allowed to be an arbitrary expression? And, similarly, foo in foo.bar? And in those cases, the repr of those parts is worse than useless (you'd get the contents of dispatch instead of its name), so there's really no alternative. Also, your suggestion means that this will assert: dispatch = {} def dispatch["foo"](): pass assert dispatch['foo'].__name__ == 'dispatch["foo"]' ? will assert, so it's a pretty weak DWIM anyway. Alexander's solution doesn't have any of those problems. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Oct 27 00:35:34 2015 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 27 Oct 2015 04:35:34 +0000 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: References: <562DC1D8.8060003@sdamon.com> Message-ID: <562EFF16.6000602@mrabarnett.plus.com> On 2015-10-26 07:27, Ian Kelly wrote: >> On Sun, Oct 25, 2015 at 11:02 PM, Alexander Walters >> wrote: >>> >>> In my code, I write a lot of dispatch dictionaries (for lack of a switch >>> statement, but I will not hold my breath for that). In trying to make >>> writing these dictionaries less annoying, I tend to use many lambdas. I can >>> let you guess at what problems that has resulted in. Of course, the >>> preferred way to write such dictionaries is by using a regular function, and >>> adding that function to a dictionary. This isn't exactly a problem - it >>> works, and works well, but it is annoying to write, and leaves artifacts of >>> those functions in module scope. I propose a little bit of sugar to make >>> this a little less annoying. >>> >>> If `def` is allowed to assign to anything (anything that is legal at the >>> left hand side of an = in that scope), annoying artifacts go away. The >>> syntax I propose should be backwards compatible. >>> >>> ``` >>> dispatch = {} >>> >>> def dispatch['foo'](bar): >>> return bar * bar >>> ``` > > What about: > > def foo(bar)[baz](x): > return x > > This seems like it would complicate parsing as the parser can't be > sure whether (bar) is a parameter list or an argument list until it > reaches the following [baz]. > Would they be simpler to parse if they were: def (bar) as dispatch['foo']: return bar * bar and: def (x) as foo(bar)[baz]: return x ? In these instances, if there's no name then it should have the "as ..." part; if there _is_ a name, then it shouldn't have the "as ..." part. It's possible that: def foo(bar): return x could just be a shorter way of writing: def (bar) as foo: return x From rosuav at gmail.com Tue Oct 27 00:29:20 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Oct 2015 15:29:20 +1100 Subject: [Python-ideas] Allowing def to assign to anything In-Reply-To: References: <562DC1D8.8060003@sdamon.com> <562DC6FA.3060309@sdamon.com> <9AA8DF99-0A00-478B-91C4-15363CCB480A@yahoo.com> Message-ID: On Tue, Oct 27, 2015 at 2:48 PM, Andrew Barnert wrote: > The paragraph you're replying to was about the naming for "def foo.bar():", > which Alexander had suggested would work differently from "def > foo['bar']():" (which he later retracted). So I'm not sure why you're > proposing a different alternative for "def foo['bar']():" in response to > these paragraphs. There've been some weirdnesses in how emails get through, and I'm not sure which previous emails I'd actually received at the time when I wrote that. Hopefully it's just a temporary outage that's now cured. If my post makes no sense in the full context of the thread, feel free to ignore it. ChrisA From storchaka at gmail.com Tue Oct 27 04:33:12 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 Oct 2015 10:33:12 +0200 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> <85pp01mhjn.fsf@benfinney.id.au> Message-ID: On 27.10.15 04:26, Chris Angelico wrote: > Fortunately, it's easy to write a function that validates a regex. > > def is_regex(s): > try: re.compile(s) > except re.error: return False > return True re.compile() also can raise OverflowError ('.{,9999999999}') and ValueError ('(?ua)'). From storchaka at gmail.com Tue Oct 27 13:45:37 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 Oct 2015 19:45:37 +0200 Subject: [Python-ideas] Specifying constants for functions Message-ID: There is known trick to optimize a function: def foo(x, y=0, len=len, pack=struct.pack, maxsize=1< References: Message-ID: <562FBA92.1060300@gmail.com> Serhiy, On 2015-10-27 1:45 PM, Serhiy Storchaka wrote: > There is known trick to optimize a function: > > def foo(x, y=0, len=len, pack=struct.pack, maxsize=1< ... > > It has a side effect: change function's signature. Would be nice to > have a way to set function's local variables at creation time without > affecting a signature. I see this a lot in all kinds of code. In my experience it doesn't actually speed things up in a measurable way. Is the below code really much slower? def foo(x, y=0): pack=struct.pack maxsize=1< References: Message-ID: On 27.10.15 19:45, Serhiy Storchaka wrote: > There is known trick to optimize a function: > > def foo(x, y=0, len=len, pack=struct.pack, maxsize=1< ... > > It has a side effect: change function's signature. Would be nice to have > a way to set function's local variables at creation time without > affecting a signature. > > Possible syntax (I'm not sure what is better): > > 1. Similar to "global" and "nonlocal" declarations with optional > initializer. > > def foo(x, y=0): > const len > const pack=struct.pack, maxsize=1< ... > > 2. Same as 1, but using "as" instead of "=". > > def foo(x, y=0): > uses len, struct.pack as pack > uses 1< ... > > 3. Declaration is moved to function header. The keyword "given" is > inspired by PEP 3150. > > def foo(x, y=0) given len=len, pack=struct.pack, maxsize=1< ... > > 4. Declaration is moved out of the function. The advantage is that bound > names can be used to evaluate default values for actual parameters (it > is useful to implement sentinel default value), and all expression are > evaluated in natural order. > > using len, struct.pack as pack, 1< def foo(x, y=0): > ... > > 5. The least wordy syntax. No new keyword needed. > > def foo(x, y=0; len=len, pack=struct.pack, maxsize=1< ... > > All above examples would be roughly equivalent to the following code: > > def create(len=len, pack=struct.pack, maxsize=1< def foo(x, y=0): > ... > return foo > tmp = create() > def foo(x, y=0): > pass > update_wrapper(tmp, foo) > foo = tmp > del create, tmp > > This feature is rather ideologically opposite to Victor's approach. Oh, only after sending my message I had read Scott's message about the asconstants decorator, that does just this optimization [1]. With this decorator above example can be written as: from codetransformer.transformers import asconstants @asconstants(len=len, pack=struct.pack, maxsize=1< References: <562FBA92.1060300@gmail.com> Message-ID: IIRC it's an old micro-optimization; this was a common idiom at Zope. But I think it's way overused -- people believe it works so they do it all the time, even for code that's not performance sensitive, just because it's become a habit. (Like "register" in C in the '80s.) On Tue, Oct 27, 2015 at 10:55 AM, Yury Selivanov wrote: > Serhiy, > > On 2015-10-27 1:45 PM, Serhiy Storchaka wrote: > >> There is known trick to optimize a function: >> >> def foo(x, y=0, len=len, pack=struct.pack, maxsize=1<> ... >> >> It has a side effect: change function's signature. Would be nice to have >> a way to set function's local variables at creation time without affecting >> a signature. >> > > I see this a lot in all kinds of code. In my experience it doesn't > actually speed things up in a measurable way. > > Is the below code really much slower? > > def foo(x, y=0): > pack=struct.pack > maxsize=1< #CODE > > If the #CODE is a tight long-running loop - then no, because the loop will > probably run much longer than an extra attribute lookup + one extra bit > shift on each "foo()" call. And if there is no tight loop - then you won't > probably notice those optimizations anyways. > > I think that adding a "const" statement deserves some discussion, but not > from the standpoint of micro-optimizations. > > Thanks, > Yury > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Oct 27 14:35:24 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 27 Oct 2015 19:35:24 +0100 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: References: <562FBA92.1060300@gmail.com> Message-ID: <562FC3EC.3050706@mail.de> I totally agree here. On 27.10.2015 19:18, Guido van Rossum wrote: > IIRC it's an old micro-optimization; this was a common idiom at Zope. > But I think it's way overused -- people believe it works so they do it > all the time, even for code that's not performance sensitive, just > because it's become a habit. (Like "register" in C in the '80s.) > > On Tue, Oct 27, 2015 at 10:55 AM, Yury Selivanov > > wrote: > > Serhiy, > > On 2015-10-27 1:45 PM, Serhiy Storchaka wrote: > > There is known trick to optimize a function: > > def foo(x, y=0, len=len, pack=struct.pack, maxsize=1< ... > > It has a side effect: change function's signature. Would be > nice to have a way to set function's local variables at > creation time without affecting a signature. > > > I see this a lot in all kinds of code. In my experience it > doesn't actually speed things up in a measurable way. > > Is the below code really much slower? > > def foo(x, y=0): > pack=struct.pack > maxsize=1< #CODE > > If the #CODE is a tight long-running loop - then no, because the > loop will probably run much longer than an extra attribute lookup > + one extra bit shift on each "foo()" call. And if there is no > tight loop - then you won't probably notice those optimizations > anyways. > > I think that adding a "const" statement deserves some discussion, > but not from the standpoint of micro-optimizations. > > Thanks, > Yury > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Oct 27 14:36:16 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 27 Oct 2015 20:36:16 +0200 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <562FBA92.1060300@gmail.com> References: <562FBA92.1060300@gmail.com> Message-ID: On 27.10.15 19:55, Yury Selivanov wrote: > I see this a lot in all kinds of code. In my experience it doesn't > actually speed things up in a measurable way. > > I think that adding a "const" statement deserves some discussion, but > not from the standpoint of micro-optimizations. Agree, we slowly gots rid of such microoptimizations. Seems that builtins and globals lookup is much faster in current CPython than was when such microoptimizations were added. Yet one application of this trick is binding globals and module's members in __del__ methods and functions that can be called at shutdown time after module's content cleared. But this is very limited application. From breamoreboy at yahoo.co.uk Tue Oct 27 14:51:42 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 27 Oct 2015 18:51:42 +0000 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <562FBA92.1060300@gmail.com> References: <562FBA92.1060300@gmail.com> Message-ID: On 27/10/2015 17:55, Yury Selivanov wrote: > Serhiy, > > On 2015-10-27 1:45 PM, Serhiy Storchaka wrote: >> There is known trick to optimize a function: >> >> def foo(x, y=0, len=len, pack=struct.pack, maxsize=1<> ... >> >> It has a side effect: change function's signature. Would be nice to >> have a way to set function's local variables at creation time without >> affecting a signature. > > I see this a lot in all kinds of code. In my experience it doesn't > actually speed things up in a measurable way. > > Is the below code really much slower? > > def foo(x, y=0): > pack=struct.pack > maxsize=1< #CODE > > If the #CODE is a tight long-running loop - then no, because the loop > will probably run much longer than an extra attribute lookup + one extra > bit shift on each "foo()" call. And if there is no tight loop - then > you won't probably notice those optimizations anyways. > > I think that adding a "const" statement deserves some discussion, but > not from the standpoint of micro-optimizations. > > Thanks, > Yury > From my very naive perspective I'd have thought that the only real difference between the two implementations is that Yury's has the optimization hard coded within the function body, while Serhiy's allows you to override the hard coded defaults at run time. Am I hot, warm, tepid, cold or approaching 0 Kelvin? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From bruce at leban.us Tue Oct 27 15:02:25 2015 From: bruce at leban.us (Bruce Leban) Date: Tue, 27 Oct 2015 12:02:25 -0700 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: References: <562FBA92.1060300@gmail.com> Message-ID: On Tue, Oct 27, 2015 at 11:18 AM, Guido van Rossum wrote: > IIRC it's an old micro-optimization; this was a common idiom at Zope. But > I think it's way overused -- people believe it works so they do it all the > time, even for code that's not performance sensitive, just because it's > become a habit. (Like "register" in C in the '80s.) > > On Tue, Oct 27, 2015 at 10:55 AM, Yury Selivanov > wrote: > >> Serhiy, >> >> On 2015-10-27 1:45 PM, Serhiy Storchaka wrote: >> >>> There is known trick to optimize a function: >>> >>> def foo(x, y=0, len=len, pack=struct.pack, maxsize=1<>> ... >> >> It's not a micro-optimization in this case: def foo(x, SENTINEL=object()): ... I don't like mangling function signatures to do this. What I really want is the equivalent of C's static here: def foo(x): static SENTINEL = object() ... This has two important semantics: (1) the scope of the SENTINEL variable is limited to the function foo; (2) it is only initialized once. Hosting the value out of the function into a decorator like @asconstants is fine. --- Bruce Check out my new puzzle book: http://J.mp/ingToConclusions Get it free here: http://J.mp/ingToConclusionsFree (available on iOS) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ian.g.kelly at gmail.com Tue Oct 27 14:59:18 2015 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 27 Oct 2015 12:59:18 -0600 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: References: <562FBA92.1060300@gmail.com> Message-ID: On Tue, Oct 27, 2015 at 12:51 PM, Mark Lawrence wrote: > On 27/10/2015 17:55, Yury Selivanov wrote: >> >> Serhiy, >> >> On 2015-10-27 1:45 PM, Serhiy Storchaka wrote: >>> >>> There is known trick to optimize a function: >>> >>> def foo(x, y=0, len=len, pack=struct.pack, maxsize=1<>> ... >>> >>> It has a side effect: change function's signature. Would be nice to >>> have a way to set function's local variables at creation time without >>> affecting a signature. >> >> >> I see this a lot in all kinds of code. In my experience it doesn't >> actually speed things up in a measurable way. >> >> Is the below code really much slower? >> >> def foo(x, y=0): >> pack=struct.pack >> maxsize=1<> #CODE >> >> If the #CODE is a tight long-running loop - then no, because the loop >> will probably run much longer than an extra attribute lookup + one extra >> bit shift on each "foo()" call. And if there is no tight loop - then >> you won't probably notice those optimizations anyways. >> >> I think that adding a "const" statement deserves some discussion, but >> not from the standpoint of micro-optimizations. >> >> Thanks, >> Yury >> > > From my very naive perspective I'd have thought that the only real > difference between the two implementations is that Yury's has the > optimization hard coded within the function body, while Serhiy's allows you > to override the hard coded defaults at run time. Am I hot, warm, tepid, > cold or approaching 0 Kelvin? Yury's code has to look up the global value of struct, and then get its pack attribute, every time the function is called. Serhiy's code only does this once, when the function is created, and when it is called the local value is loaded directly from the function defaults, which amounts to a single tuple lookup rather than two sequential dict lookups. From Andy.Henshaw at gtri.gatech.edu Tue Oct 27 15:16:31 2015 From: Andy.Henshaw at gtri.gatech.edu (Henshaw, Andy) Date: Tue, 27 Oct 2015 19:16:31 +0000 Subject: [Python-ideas] PEP 8: raw strings & regular expressions In-Reply-To: References: <562848C2.4090509@gmail.com> <20151022171703.797395e1@anarchist.wooz.org> <562969B4.5080406@gmail.com> <562A7F1D.90108@nedbatchelder.com> <562DD4F3.5000901@sdamon.com> <562E0F92.7050400@nedbatchelder.com> <72B2B3B1-3F0D-4CB3-9B64-5B3F7114CA2C@yahoo.com> <85pp01mhjn.fsf@benfinney.id.au> Message-ID: <2d7acb865d5b4d2ca18ee9ac31b5e4eb@apatlisdmail01.core.gtri.org> On Monday, October 26, 2015 10:26 PM, Chris Angelico wrote > > On Tue, Oct 27, 2015 at 8:44 AM, Ben Finney > wrote: > > You seem to be seeking something else: a pattern that matches all > > valid regex patterns, *and* will never match any string that is not a > > valid regex pattern. The latter is rather more difficult. > > "Rather more difficult" may be an understatement... Indeed, in fact I think that G?del's Incompleteness Theorem applies here. Andy Henshaw From yselivanov.ml at gmail.com Tue Oct 27 15:47:02 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 27 Oct 2015 15:47:02 -0400 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: References: <562FBA92.1060300@gmail.com> Message-ID: <562FD4B6.7060105@gmail.com> On 2015-10-27 3:02 PM, Bruce Leban wrote: > It's not a micro-optimization in this case: > > def foo(x, SENTINEL=object()): > ... > > > I don't like mangling function signatures to do this. What I really > want is the equivalent of C's static here: > > def foo(x): > static SENTINEL = object() > ... > > Why not SENTINEL=object() def foo(x): ... ? That's how I do it usually. Yury From rymg19 at gmail.com Tue Oct 27 16:15:24 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 27 Oct 2015 15:15:24 -0500 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: References: Message-ID: Making a decorator is easy: def opt(**kw): def func(f): globals().update(dis.opmap) # Someone's going to kill me for this... code = list(f.__code__.co_code) offs = 0 varnames = list(f.__code__.co_varnames) nlocals = f.__code__.co_nlocals names = list(f.__code__.co_names) globls = f.__globals__ mapping = {} anon_count = 0 for k, v in kw.items(): if k in names: try: i = names.index(k) except ValueError: raise ValueError('variable %r must be a global' % k) from None anon_var = '$__varopt__%d$' % anon_count anon_count += 1 names[i] = anon_var globls[anon_var] = v assert k not in varnames, '%r is both global and local' % k varnames.append(k) nlocals += 1 mapping[i] = nlocals-1 code[:0] = [ LOAD_GLOBAL, i, 0, STORE_FAST, nlocals-1, 0, ] offs += 6 else: raise ValueError('variable %r is not a global' % k) i = offs while i < len(code): if code[i] in dis.hasjabs: code[i+1] += offs elif code[i] in (LOAD_GLOBAL, STORE_GLOBAL) and code[i+1] in mapping: code[i] = LOAD_FAST if code[i] == LOAD_GLOBAL else STORE_FAST code[i+1] = mapping[code[i+1]] i += 3 if code[i] > dis.HAVE_ARGUMENT else 1 rescode = types.CodeType(f.__code__.co_argcount, f.__code__.co_kwonlyargcount, nlocals, f.__code__.co_stacksize, f.__code__.co_flags, bytes(code), f.__code__.co_consts, tuple(names), tuple(varnames), f.__code__.co_filename, f.__code__.co_name, f.__code__.co_firstlineno, f.__code__.co_lnotab, f.__code__.co_freevars, f.__code__.co_cellvars) return types.FunctionType(rescode, globls, f.__name__, f.__defaults__, f.__closure__) return func On Tue, Oct 27, 2015 at 12:45 PM, Serhiy Storchaka wrote: > There is known trick to optimize a function: > > def foo(x, y=0, len=len, pack=struct.pack, maxsize=1< ... > > It has a side effect: change function's signature. Would be nice to have a > way to set function's local variables at creation time without affecting a > signature. > > Possible syntax (I'm not sure what is better): > > 1. Similar to "global" and "nonlocal" declarations with optional > initializer. > > def foo(x, y=0): > const len > const pack=struct.pack, maxsize=1< ... > > 2. Same as 1, but using "as" instead of "=". > > def foo(x, y=0): > uses len, struct.pack as pack > uses 1< ... > > 3. Declaration is moved to function header. The keyword "given" is > inspired by PEP 3150. > > def foo(x, y=0) given len=len, pack=struct.pack, maxsize=1< ... > > 4. Declaration is moved out of the function. The advantage is that bound > names can be used to evaluate default values for actual parameters (it is > useful to implement sentinel default value), and all expression are > evaluated in natural order. > > using len, struct.pack as pack, 1< def foo(x, y=0): > ... > > 5. The least wordy syntax. No new keyword needed. > > def foo(x, y=0; len=len, pack=struct.pack, maxsize=1< ... > > All above examples would be roughly equivalent to the following code: > > def create(len=len, pack=struct.pack, maxsize=1< def foo(x, y=0): > ... > return foo > tmp = create() > def foo(x, y=0): > pass > update_wrapper(tmp, foo) > foo = tmp > del create, tmp > > This feature is rather ideologically opposite to Victor's approach. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Tue Oct 27 16:22:22 2015 From: Nikolaus at rath.org (Nikolaus Rath) Date: Tue, 27 Oct 2015 13:22:22 -0700 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <562FD4B6.7060105@gmail.com> (Yury Selivanov's message of "Tue, 27 Oct 2015 15:47:02 -0400") References: <562FBA92.1060300@gmail.com> <562FD4B6.7060105@gmail.com> Message-ID: <87io5sxds1.fsf@thinkpad.rath.org> On Oct 27 2015, Yury Selivanov wrote: > On 2015-10-27 3:02 PM, Bruce Leban wrote: >> It's not a micro-optimization in this case: >> >> def foo(x, SENTINEL=object()): >> ... >> >> >> I don't like mangling function signatures to do this. What I really >> want is the equivalent of C's static here: >> >> def foo(x): >> static SENTINEL = object() >> ... >> >> > > Why not > > SENTINEL=object() > def foo(x): > ... > > ? Because it leaks into the enclosing scope. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From yselivanov.ml at gmail.com Tue Oct 27 16:30:41 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 27 Oct 2015 16:30:41 -0400 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <87io5sxds1.fsf@thinkpad.rath.org> References: <562FBA92.1060300@gmail.com> <562FD4B6.7060105@gmail.com> <87io5sxds1.fsf@thinkpad.rath.org> Message-ID: <562FDEF1.8050303@gmail.com> On 2015-10-27 4:22 PM, Nikolaus Rath wrote: > Because it leaks into the enclosing scope. Leaks what? Name? Name it _SENTINEL then. Yury From srkunze at mail.de Tue Oct 27 17:50:16 2015 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 27 Oct 2015 22:50:16 +0100 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <562FDEF1.8050303@gmail.com> References: <562FBA92.1060300@gmail.com> <562FD4B6.7060105@gmail.com> <87io5sxds1.fsf@thinkpad.rath.org> <562FDEF1.8050303@gmail.com> Message-ID: <562FF198.5080708@mail.de> On 27.10.2015 21:30, Yury Selivanov wrote: > On 2015-10-27 4:22 PM, Nikolaus Rath wrote: >> Because it leaks into the enclosing scope. > > Leaks what? > Name? Name it _SENTINEL then. I think because it's missing the relationship with foo. So, better do this foo._sentinel = object() It's coupled to foo and moved out of the outer scope. Best, Sven From steve at pearwood.info Tue Oct 27 19:36:50 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Oct 2015 10:36:50 +1100 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <562FDEF1.8050303@gmail.com> References: <562FBA92.1060300@gmail.com> <562FD4B6.7060105@gmail.com> <87io5sxds1.fsf@thinkpad.rath.org> <562FDEF1.8050303@gmail.com> Message-ID: <20151027233650.GB3813@ando.pearwood.info> On Tue, Oct 27, 2015 at 04:30:41PM -0400, Yury Selivanov wrote: > > > On 2015-10-27 4:22 PM, Nikolaus Rath wrote: > >Because it leaks into the enclosing scope. > > Leaks what? > Name? Name it _SENTINEL then. It is still in the enclosing scope. Bruce is right that what we really want is something like "static". Lacking that feature, one work-around is "make it a global variable, and give it a leading underscore", but that's just a work-around. Consider: _SENTINEL = object() # Later: def spam(): ... def eggs(): ... def cheese(): ... def aardvark(): ... As the module user, you can see the leading underscore in _SENTINEL and immediately forget all about it. But as the module maintainer, you cannot ignore _SENTINEL. Leading underscore or not, it is still part of the implementation, and maintenance is all about the implementation. Which of the functions uses _SENTINEL? You can assume that *at least* one function uses it, but it may be more. Is it safe to rename it? Change it's value? To the maintainer, _SENTINEL is just another global variable, with all the disadvantages that has. Better to put it inside the function, so the maintainer knows that it is local to the function: def spam(): SENTINEL = object() ... That's not bad, but what if the value is something harder to calculate? def spam(): SENTINEL = next_prime_number(2**512) ... At the moment, Python has an obvious way to calculate a value once only (make it a module-level global), and there is an obvious way to make a value local to a function (put it inside the function body). But there's no obvious way to do both together: "Calculate this thing once only, AND make it local to this scope." The unobvious and ugly way is to put the calculation in the function declaration as a default value: def spam(SENTINEL=next_prime_number(2**512)): ... which complicates the function signature and risks errors if the caller accidentally calls the function with too many arguments. -- Steve From steve at pearwood.info Tue Oct 27 19:44:18 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Oct 2015 10:44:18 +1100 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: References: Message-ID: <20151027234418.GC3813@ando.pearwood.info> On Tue, Oct 27, 2015 at 03:15:24PM -0500, Ryan Gonzalez wrote: > Making a decorator is easy: [snip 54 lines of hacking the internals of function and code objects] I like the way you said that with a straight face. :-) -- Steve From guido at python.org Tue Oct 27 19:47:03 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Oct 2015 16:47:03 -0700 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <20151027233650.GB3813@ando.pearwood.info> References: <562FBA92.1060300@gmail.com> <562FD4B6.7060105@gmail.com> <87io5sxds1.fsf@thinkpad.rath.org> <562FDEF1.8050303@gmail.com> <20151027233650.GB3813@ando.pearwood.info> Message-ID: On Tue, Oct 27, 2015 at 4:36 PM, Steven D'Aprano wrote: > On Tue, Oct 27, 2015 at 04:30:41PM -0400, Yury Selivanov wrote: > > > > > > On 2015-10-27 4:22 PM, Nikolaus Rath wrote: > > >Because it leaks into the enclosing scope. > > > > Leaks what? > > Name? Name it _SENTINEL then. > > It is still in the enclosing scope. > > Bruce is right that what we really want is something like "static". > Lacking that feature, one work-around is "make it a global variable, and > give it a leading underscore", but that's just a work-around. > > Consider: > > _SENTINEL = object() > > # Later: > def spam(): ... > def eggs(): ... > def cheese(): ... > def aardvark(): ... > > > As the module user, you can see the leading underscore in _SENTINEL > and immediately forget all about it. But as the module maintainer, you > cannot ignore _SENTINEL. Leading underscore or not, it is still part of > the implementation, and maintenance is all about the implementation. > > Which of the functions uses _SENTINEL? You can assume that *at least* > one function uses it, but it may be more. Is it safe to rename it? > Change it's value? To the maintainer, _SENTINEL is just another global > variable, with all the disadvantages that has. > > Better to put it inside the function, so the maintainer knows that it is > local to the function: > > def spam(): > SENTINEL = object() > ... > > > That's not bad, but what if the value is something harder to calculate? > > def spam(): > SENTINEL = next_prime_number(2**512) > ... > > > At the moment, Python has an obvious way to calculate a value once only > (make it a module-level global), and there is an obvious way to make a > value local to a function (put it inside the function body). But there's > no obvious way to do both together: > > "Calculate this thing once only, AND make it local to this scope." > > The unobvious and ugly way is to put the calculation in the function > declaration as a default value: > > def spam(SENTINEL=next_prime_number(2**512)): > ... > > > which complicates the function signature and risks errors if the caller > accidentally calls the function with too many arguments. Python intentionally doesn't have this feature, and the argument you present sounds pretty weak (compared to the compelling argument *against* abusing the default argument syntax -- which also has a maintenance cost, as you have to explain it over and over). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Tue Oct 27 20:44:47 2015 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 28 Oct 2015 09:44:47 +0900 Subject: [Python-ideas] ssl module: validated certificate chain Message-ID: All, I?m currently working on adding support for HPKP to the Requests and urllib3 modules. HPKP (HTTP Public Key Pinning), specified in RFC 7469, is an extension to HTTP that allows a web server to specify a whitelist of public keys that are valid for TLS certificates on that domain. This prevents a rogue certificate authority from issuing a certificate that would be trusted by a browser and would allow a man-in-the-middle attack on a domain (as happened to Google in 2013[0]). Right now, the draft version of the support I have will only work when you use PyOpenSSL for your TLS needs, not the standard library. This is because to get HPKP to work I need access to the validated certificate chain: that is, the certificate chain that OpenSSL has built and validated for the TLS connection. I also need to be able to work with those certificates in order to extract their public keys. The standard library?s ssl module does not expose any of this functionality. I want to get an idea of whether people believe it should be possible to access this functionality in the standard library. It would require the following things from the standard library: 1. The ability to access the validated certificate chain. This requires saving off the certificate each time the OpenSSL verify callback is called. This is an easy enough change to make. 2. The ability to extract the public key from the saved certificates. This could be done by extending the logic used for getpeercert() to provide a DER-encoded ASN.1 representation of the public key in the dictionary, and then using that representation for each cert in the peer cert chain. The motivation for making this available in the standard library would be pip. Right now python.org and all its subdomains (including pypi.python.org) are HPKP-enabled. Making this support available in the standard library would ensure that all pip installations are safe from man-in-the-middle attacks on its packaging infrastructure. Without it, a number of third-party packages would be required to add this security. I?m happy to do the work required to provide this functionality, but I?d only like to start work if people believe there?s a likelihood of getting it merged. Thanks, Cory [0]: https://nakedsecurity.sophos.com/2013/01/08/the-turktrust-ssl-certificate-fiasco-what-happened-and-what-happens-next/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tjreedy at udel.edu Tue Oct 27 20:57:31 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Oct 2015 20:57:31 -0400 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: <20151027233650.GB3813@ando.pearwood.info> References: <562FBA92.1060300@gmail.com> <562FD4B6.7060105@gmail.com> <87io5sxds1.fsf@thinkpad.rath.org> <562FDEF1.8050303@gmail.com> <20151027233650.GB3813@ando.pearwood.info> Message-ID: On 10/27/2015 7:36 PM, Steven D'Aprano wrote: > The unobvious and ugly way is to put the calculation in the function > declaration as a default value: > > def spam(SENTINEL=next_prime_number(2**512)): > ... > > > which complicates the function signature and risks errors if the caller > accidentally calls the function with too many arguments. Making constant names keyword only avoids the 'too many arguments' problem. >>> def spam(*, _SETINAL=object): pass >>> spam(1) Traceback (most recent call last): File "", line 1, in spam(1) TypeError: spam() takes 0 positional arguments but 1 was given Writing 'spam(_SETINAL=3)' would not be an accident ;-) -- Terry Jan Reedy From victor.stinner at gmail.com Tue Oct 27 22:48:41 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 28 Oct 2015 11:48:41 +0900 Subject: [Python-ideas] Specifying constants for functions In-Reply-To: References: Message-ID: Hi, 2015-10-28 2:45 GMT+09:00 Serhiy Storchaka : > There is known trick to optimize a function: > > def foo(x, y=0, len=len, pack=struct.pack, maxsize=1< ... Yeah, it can show a speedup on a micro-benchmark. Probably not a macro benchmark. As it was said in other answers, this hack is also abused for bad reasons. This hack is mainly used in the stdlib to keep symbols alive during Python shutdown to be able to cleanup properly objects. Just one example from Lib/subprocess.py of Python 3.6: "def __del__(self, _maxsize=sys.maxsize):". I guess that sys.maxsize symbol is removed or set to None during Python shutdown. So depending on the order in which modules are cleared (subprocess,sys or sys,subprocess), the __del__() method may fail or may not fail without the "_maxsize=sys.maxsize" hack. I would appreciate a syntax to not change the function signature, even if this hack is mostly used in destructors and destructors must *not* be called explicitly. > This feature is rather ideologically opposite to Victor's approach. I disagree, it's not incompatible with my FAT Python project. In some cases, we may still see speedup if you combine copying globals to locals and using FAT Python optimizations. My idea is more to optimize code without having to modify manually the code to optimize it. Using FAT Python, you can implement an optimizer producing code like: --- import builtins def f(data): lengths = [] for item in data: lengths.append(len(item)) return lengths def f_copy_globals(data, _len=len): lengths = [] for item in data: lengths.append("fast: %s" % _len(item)) # add "fast" to ensure that we call the "fast" function return lengths i = f.specialize(f_copy_globals) f.add_dict_guard(i, builtins.__dict__, 'len') f.add_dict_guard(i, globals(), 'len') # test specialized function with "fast" _len local symbol data = ["abc", list(range(5))] print(f(data)) # test with a mocked len() builtin function builtins.len = lambda obj: 10 data = ["abc", list(range(5))] print(f(data)) --- Output: --- ['fast: 3', 'fast: 5'] [10, 10] --- This optimization doesn't respect strictly Python semantic because the len() builtin function can be modified during two loop iterations in a different Python thread. In some cases, it can be worth to optimize the function and doesn't respect stricly the Python semantic. As it was discussed in the FAT Python thread, depending on your use case, you may or may not allow some optimizations. Note: This example doesn't work with my current implementation of FAT Python, because f() and f_copy_globals() don't have the same default values for parameters. You can test with "def f(data, _len=len):". I have to modify FAT Python to support this example. There is also a bug if the specialized function uses a free variable, but not the original function. Again, it should enhance FAT Python to support this case. Victor From greg at krypto.org Wed Oct 28 20:21:52 2015 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 29 Oct 2015 00:21:52 +0000 Subject: [Python-ideas] ssl module: validated certificate chain In-Reply-To: References: Message-ID: I'm in favor of this. I don't know if we'd ever land this kind of change (new feature) in existing stable 2.7 or 3.4/3.5 releases, but for strictly security related things like this we have done so in the past. If nothing else, there is no reason not to have this in 3.6. It is good for the future. -gps On Tue, Oct 27, 2015 at 5:45 PM Cory Benfield wrote: > All, > > I?m currently working on adding support for HPKP to the Requests and > urllib3 modules. HPKP (HTTP Public Key Pinning), specified in RFC 7469, is > an extension to HTTP that allows a web server to specify a whitelist of > public keys that are valid for TLS certificates on that domain. This > prevents a rogue certificate authority from issuing a certificate that > would be trusted by a browser and would allow a man-in-the-middle attack on > a domain (as happened to Google in 2013[0]). > > Right now, the draft version of the support I have will only work when you > use PyOpenSSL for your TLS needs, not the standard library. This is because > to get HPKP to work I need access to the validated certificate chain: that > is, the certificate chain that OpenSSL has built and validated for the TLS > connection. I also need to be able to work with those certificates in order > to extract their public keys. The standard library?s ssl module does not > expose any of this functionality. > > I want to get an idea of whether people believe it should be possible to > access this functionality in the standard library. It would require the > following things from the standard library: > > 1. The ability to access the validated certificate chain. This requires > saving off the certificate each time the OpenSSL verify callback is called. > This is an easy enough change to make. > 2. The ability to extract the public key from the saved certificates. This > could be done by extending the logic used for getpeercert() to provide a > DER-encoded ASN.1 representation of the public key in the dictionary, and > then using that representation for each cert in the peer cert chain. > > The motivation for making this available in the standard library would be > pip. Right now python.org and all its subdomains (including > pypi.python.org) are HPKP-enabled. Making this support available in the > standard library would ensure that all pip installations are safe from > man-in-the-middle attacks on its packaging infrastructure. Without it, a > number of third-party packages would be required to add this security. > > I?m happy to do the work required to provide this functionality, but I?d > only like to start work if people believe there?s a likelihood of getting > it merged. > > Thanks, > > Cory > > > [0]: > https://nakedsecurity.sophos.com/2013/01/08/the-turktrust-ssl-certificate-fiasco-what-happened-and-what-happens-next/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Oct 30 21:09:00 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Oct 2015 21:09:00 -0400 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module Message-ID: This idea results from issue of user files shadowing stdlib files on import. There was a thread on pydev about this yesterday. There is also an opposite issue of builtin modules shadowing user files. The keyword module provides kwlist and iskeyword function. One use of kwlist is used in some other stdlib modules and can be used by syntax highlighters (as in IDLE). Kwlist is updated by the main function. A module module would have at least liblist and islibmodule function. Liblist would contain all directories with __init__.py and all .py files. (I don't think files within package directories should be included, as there is no direct shadowing problem.) A python oriented editor could then warn on save requests "This name matches a stdlib name in /Lib. If you run python in this directory, you will not be able to import the stdlib module. Continue?". The module should also have binlist and isbinmodule for builtin modules. (I do not know how to get such a list. If necessary, an api could be added.) An editor could than warn "This name matches a builtin stdlib name. You will not be able to import this file. Continue?". -- Terry Jan Reedy From guido at python.org Fri Oct 30 21:19:54 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Oct 2015 18:19:54 -0700 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: Message-ID: There's sys.builtin_module_names which returns the names of the hardcoded builtin modules. Dynamically loaded modules can be found by searching sys.path in the usual way -- importlib shoul know. I wonder if just asking importlib whether it can locate a given module would be enough? On Fri, Oct 30, 2015 at 6:09 PM, Terry Reedy wrote: > This idea results from issue of user files shadowing stdlib files on > import. There was a thread on pydev about this yesterday. There is also > an opposite issue of builtin modules shadowing user files. > > The keyword module provides kwlist and iskeyword function. One use of > kwlist is used in some other stdlib modules and can be used by syntax > highlighters (as in IDLE). Kwlist is updated by the main function. > > A module module would have at least liblist and islibmodule function. > Liblist would contain all directories with __init__.py and all .py files. > (I don't think files within package directories should be included, as > there is no direct shadowing problem.) A python oriented editor could then > warn on save requests "This name matches a stdlib name in /Lib. If you run > python in this directory, you will not be able to import the stdlib > module. Continue?". > > The module should also have binlist and isbinmodule for builtin modules. > (I do not know how to get such a list. If necessary, an api could be > added.) An editor could than warn "This name matches a builtin stdlib > name. You will not be able to import this file. Continue?". > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Oct 31 01:56:33 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 31 Oct 2015 01:56:33 -0400 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: Message-ID: On 10/30/2015 9:19 PM, Guido van Rossum wrote: > There's sys.builtin_module_names which returns the names of the > hardcoded builtin modules. Great. With this solved, I opened an issue for IDLE. https://bugs.python.org/issue25522 > Dynamically loaded modules can be found by > searching sys.path in the usual way -- importlib shoul know. I wonder if > just asking importlib whether it can locate a given module would be enough? The default search order is stdlib builtins, local user files, /lib files, so the shadowing issue the opposite for builtin and /lib modules. Hence a different message is needed. > On Fri, Oct 30, 2015 at 6:09 PM, Terry Reedy > > wrote: > > This idea results from issue of user files shadowing stdlib files on > import. There was a thread on pydev about this yesterday. There is > also an opposite issue of builtin modules shadowing user files. > > The keyword module provides kwlist and iskeyword function. One use > of kwlist is used in some other stdlib modules and can be used by > syntax highlighters (as in IDLE). Kwlist is updated by the main > function. > > A module module would have at least liblist and islibmodule > function. Liblist would contain all directories with __init__.py and > all .py files. (I don't think files within package directories > should be included, as there is no direct shadowing problem.) A > python oriented editor could then warn on save requests "This name > matches a stdlib name in /Lib. If you run python in this directory, > you will not be able to import the stdlib module. Continue?". > > The module should also have binlist and isbinmodule for builtin > modules. (I do not know how to get such a list. If necessary, an > api could be added.) An editor could than warn "This name matches a > builtin stdlib name. You will not be able to import this file. > Continue?". > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy From steve at pearwood.info Sat Oct 31 04:14:51 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 31 Oct 2015 19:14:51 +1100 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: Message-ID: <20151031081450.GD10946@ando.pearwood.info> On Fri, Oct 30, 2015 at 09:09:00PM -0400, Terry Reedy wrote: > A module module would have at least liblist and islibmodule function. > Liblist would contain all directories with __init__.py and all .py > files. `liblist` cannot usefully be a statically created list, since the availability of modules on the path is dynamic: the path can change, and files can be added or removed. Nor can it be limited to .py files, since .pyc and other extensions can be imported. I don't know what `islibmodule` is supposed to do. Surely all libraries are modules? > (I don't think files within package directories should be > included, as there is no direct shadowing problem.) Sounds like you want a private "fix_shadowing_of_modules_for_IDLE" module, rather than a general-purpose "module" module. For a general purpose "module" module, whether or not there is a shadowing problem is irrelevant. By the way, it should be clear from the above that the name "module" is atrocious -- it makes it awkward to talk about the module, and it will itself be shadowed by anyone (especially beginners) who create a "module.py" file. "modtools" might be a better name. > A python oriented > editor could then warn on save requests "This name matches a stdlib name > in /Lib. If you run python in this directory, you will not be able to > import the stdlib module. Continue?". I don't think this is much of a solution to the shadowing problem. For starters, it relies on people using a specific editor. It assumes that files won't be renamed or moved outside of the editor. It depends on the user actually paying attention and reading the error message, which beginners notoriously don't do. Many people will either blindly continue (and hence shadow), or dismiss the dialog and then get into a flap that they can't save their work. > The module should also have binlist and isbinmodule for builtin modules. > (I do not know how to get such a list. If necessary, an api could be > added.) An editor could than warn "This name matches a builtin stdlib > name. You will not be able to import this file. Continue?". binlist is easy: sys.builtin_module_names. I don't know what `isbinmodule` means either. Presumably you don't actually mean "is binary module", but "is builtin module". I believe that the canonical way to do that is hasattr(module, "__file__"). -- Steve From brett at python.org Sat Oct 31 12:48:45 2015 From: brett at python.org (Brett Cannon) Date: Sat, 31 Oct 2015 16:48:45 +0000 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: Message-ID: On Fri, 30 Oct 2015 at 22:57 Terry Reedy wrote: > On 10/30/2015 9:19 PM, Guido van Rossum wrote: > > There's sys.builtin_module_names which returns the names of the > > hardcoded builtin modules. > > Great. With this solved, I opened an issue for IDLE. > https://bugs.python.org/issue25522 > > > Dynamically loaded modules can be found by > > searching sys.path in the usual way -- importlib shoul know. I wonder if > > just asking importlib whether it can locate a given module would be > enough? > > The default search order is stdlib builtins, local user files, /lib > files, so the shadowing issue the opposite for builtin and /lib modules. > Hence a different message is needed. > Quick and dirty way to use importlib is to get the location of the stdlib (os.__file__ should work since it's hard-coded in the interpreter as representing where the stdlib is) and then use importlib.find_spec() for a module name to check if the file location in the spec has the same location prefix as os or not (make sure you use absolute paths since it isn't guaranteed if you don't execute site.py). I've now seen this use case, the logging one, and the 2to3 module rename. I'm starting to wonder if there some general solution that should get added to the import machinery that can serve these cases more easily than with importers or __import__ overrides that can be tricky to get right. -Brett > > > On Fri, Oct 30, 2015 at 6:09 PM, Terry Reedy > > > > wrote: > > > > This idea results from issue of user files shadowing stdlib files on > > import. There was a thread on pydev about this yesterday. There is > > also an opposite issue of builtin modules shadowing user files. > > > > The keyword module provides kwlist and iskeyword function. One use > > of kwlist is used in some other stdlib modules and can be used by > > syntax highlighters (as in IDLE). Kwlist is updated by the main > > function. > > > > A module module would have at least liblist and islibmodule > > function. Liblist would contain all directories with __init__.py and > > all .py files. (I don't think files within package directories > > should be included, as there is no direct shadowing problem.) A > > python oriented editor could then warn on save requests "This name > > matches a stdlib name in /Lib. If you run python in this directory, > > you will not be able to import the stdlib module. Continue?". > > > > The module should also have binlist and isbinmodule for builtin > > modules. (I do not know how to get such a list. If necessary, an > > api could be added.) An editor could than warn "This name matches a > > builtin stdlib name. You will not be able to import this file. > > Continue?". > > > > -- > > Terry Jan Reedy > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido ) > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sat Oct 31 14:31:52 2015 From: steve.dower at python.org (Steve Dower) Date: Sat, 31 Oct 2015 11:31:52 -0700 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: Message-ID: Could we add a context manager to importlib (or perhaps site or sys) to temporarily disable imports from non-standard paths? I don't see any safe way to change the default behavior, but no reason we can't make it easy for applications to self-isolate. Top-posted from my Windows Phone -----Original Message----- From: "Brett Cannon" Sent: ?10/?31/?2015 9:50 To: "Terry Reedy" ; "python-ideas at python.org" Subject: Re: [Python-ideas] Add 'module' module, similar to 'keyword' module On Fri, 30 Oct 2015 at 22:57 Terry Reedy wrote: On 10/30/2015 9:19 PM, Guido van Rossum wrote: > There's sys.builtin_module_names which returns the names of the > hardcoded builtin modules. Great. With this solved, I opened an issue for IDLE. https://bugs.python.org/issue25522 > Dynamically loaded modules can be found by > searching sys.path in the usual way -- importlib shoul know. I wonder if > just asking importlib whether it can locate a given module would be enough? The default search order is stdlib builtins, local user files, /lib files, so the shadowing issue the opposite for builtin and /lib modules. Hence a different message is needed. Quick and dirty way to use importlib is to get the location of the stdlib (os.__file__ should work since it's hard-coded in the interpreter as representing where the stdlib is) and then use importlib.find_spec() for a module name to check if the file location in the spec has the same location prefix as os or not (make sure you use absolute paths since it isn't guaranteed if you don't execute site.py). I've now seen this use case, the logging one, and the 2to3 module rename. I'm starting to wonder if there some general solution that should get added to the import machinery that can serve these cases more easily than with importers or __import__ overrides that can be tricky to get right. -Brett > On Fri, Oct 30, 2015 at 6:09 PM, Terry Reedy > > wrote: > > This idea results from issue of user files shadowing stdlib files on > import. There was a thread on pydev about this yesterday. There is > also an opposite issue of builtin modules shadowing user files. > > The keyword module provides kwlist and iskeyword function. One use > of kwlist is used in some other stdlib modules and can be used by > syntax highlighters (as in IDLE). Kwlist is updated by the main > function. > > A module module would have at least liblist and islibmodule > function. Liblist would contain all directories with __init__.py and > all .py files. (I don't think files within package directories > should be included, as there is no direct shadowing problem.) A > python oriented editor could then warn on save requests "This name > matches a stdlib name in /Lib. If you run python in this directory, > you will not be able to import the stdlib module. Continue?". > > The module should also have binlist and isbinmodule for builtin > modules. (I do not know how to get such a list. If necessary, an > api could be added.) An editor could than warn "This name matches a > builtin stdlib name. You will not be able to import this file. > Continue?". > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Oct 31 18:45:15 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 31 Oct 2015 18:45:15 -0400 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: <20151031081450.GD10946@ando.pearwood.info> References: <20151031081450.GD10946@ando.pearwood.info> Message-ID: On 10/31/2015 4:14 AM, Steven D'Aprano wrote: A response to my already half-dead proposal. Literally 10 minutes after I posted it, Guido replied that half of what I wanted already existed as sys.builtin_module_names. I acknowledged that, two hours before Steven's post, by saying I would go ahead and use this tuple for IDLE. General comments: The possibility of name clashes arises because Python does not limit import to stdlib. As Steven noted, Python cannot prevent this. What it could do is search the all possible import sources with each import and report clashes before picking one. I am not proposing this. Whether python picks a user file or stdlib file when both have the same name depends on how the stdlib version is implemented. Specific comment: I limited the proposal to the stdlib because a) the stdlib is fixed for a given version of CPython on a particular OS*, and b) the reported problems of beginners that I have seen, where they are stuck on what to do, involve the stdlib. (I could have made this limitation clearer in my original first paragraph.) *Except as modules are omitted in a particular build. sys.builtin_module_names exists because the information is not otherwise exposed and because it is needed and used in several places. Revised and reduced proposal: If other people would find it useful, add all_stdlib_toplevel_module_names - builtin_module_names to sys as something equivalent to .other_stdlib_module_names or .python_coded_module_names. Anyone wanting all_toplevel_module names could add the two. Or add the latter, and let others subtract. In the meanwhile, I will adapt the code in test__all__ that creates such a list from the lib directory. -- Terry Jan Reedy From brenbarn at brenbarn.net Sat Oct 31 19:07:02 2015 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sat, 31 Oct 2015 16:07:02 -0700 Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module In-Reply-To: References: <20151031081450.GD10946@ando.pearwood.info> Message-ID: <56354996.8030807@brenbarn.net> On 2015-10-31 15:45, Terry Reedy wrote: > Revised and reduced proposal: If other people would find it useful, add > all_stdlib_toplevel_module_names - builtin_module_names to sys as > something equivalent to .other_stdlib_module_names or > .python_coded_module_names. Anyone wanting all_toplevel_module names > could add the two. Or add the latter, and let others subtract. One thing I've sometimes wondered about for Python 4000 is the idea of putting the whole standard library under a single top-level package (e.g., from stdlib import sys). This would be a big change but would reduce the surprises that can arise from stdlib modules named things like "string", "parser", etc. I see a brief mention of this in PEP 3108 but that's it. Was there more discussion of the idea? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown